---
title: "Good news about good news? The limited impacts of informing Americans about recent success in climate change mitigation"
execute:
  echo: false
  message: false
  warning: false
date: last-modified
format: pdf
author:
  - name: |
          | Olivia Chiancone, Tara Chugh, Brayant De Leon-Duarte, 
          | Andrew C. Eggers,  Benjamin Fica, Ishaan Goel,  
          | Stacia Konow, Annika Lundsgaard, Julia Margie, Zane Miller, 
          | Manuela Pinheiro, Panteleymon Semka, John Stokes, 
          | Leena Tantawy, Panthita Triamkitsawat, and Gus Waldspurger
    affiliations: University of Chicago 
    acknowledgements: |
      Eggers is Professor of Political Science at the University of Chicago. Other authors were undergraduate students in "Social Science Inquiry" at the University of Chicago in Spring of 2023. We thank Molly Offer-Westort, Anton Strezhnev, David Konisky, and Zikai Li for feedback. Pre-analysis plan ("Information effects on climate change preferences and beliefs") available at <https://osf.io/zkptf>.   
abstract: |
  Slowing the process of global warming will require sustained reductions in greenhouse gas (GHG) emissions over decades, which in turn depends on public support for decarbonization. We are beginning to see evidence of success in cutting emissions. Will wider recognition of this "good news" strengthen or soften support for further action? In contrast to previous research showing a demotivating effect of good news about climate mitigation, our pre-registered survey experiment finds no average impact of factual reports of climate progress on Americans' worries about climate change, perception of efficacy in fighting climate change, willingness to pay additional taxes, support for green industrial policy, or willingness to donate to a climate NGO, and no evidence of heterogeneous effects across relevant subgroups. Given the importance of sustained public support for decarbonization, these null results can be seen as good news for mitigation policy, but more research is needed to assess the impact of the perceived tractability of climate mitigation on support for mitigation efforts.
bibliography: climate.bib  
linestretch: 1.5

---


<!-- format:
This is the formatting we use in the working paper (requires some extra style files)
  jasa-pdf:
    keep-tex: true  
    journal:
      blinded: false
  jasa-html: default
--> 

<!-- PAP link: https://osf.io/zkptf --> 

\spacingset{1}  <!-- undoing the double spacing -->
\doublespacing  <!-- redoing in a way that does not double space the tables --> 
\setlength{\parindent}{12pt}  <!-- paragrpah indents --> 
\setlength{\parskip}{0pt} <!-- 6pt plus 2pt minus 1pt}} --> 

## Introduction

Although the global average temperature continues to rise and severe weather events have become more common in recent years due to global warming,[^ipccreport] there are also encouraging recent signs that GHG emissions can be contained and reduced. Notably, the cost of generating renewable energy from solar and wind power has steadily declined, such that utility-scale solar plants recently became the cheapest-ever way of generating energy,[^irena] and the production of renewable energy is rising quickly in many countries.  Emissions have been declining in the US and the European Union for fifteen years, and some analysts expect global emissions to start steadily declining by 2025.[^mckinsey] 

In this paper, we seek to better understand how the public reacts to these encouraging but not-yet-widely-known pieces of good news about climate change. As @hornsey2016cautionary note, learning about progress in addressing climate change could make people more supportive of further action, because the problem seems more tractable;  it could also make people *less* supportive of further action, because the problem seems less severe.  @hornsey2016cautionary conclude on the basis of a US survey experiment that the latter mechanism is more pervasive: an optimistic message about the trajectory of global emissions makes people less motivated to take action. This could be bad news for the mitigation of climate change, given that constraining further warming will require sustained public support for aggressive action even as emissions drop much further [@drews2016explains;@azevedo2020paths].[^ipccreport] Other studies find limited effects of more or less optimistic climate messages [e.g. @bernauer2016simple;@ettinger2021climate]. Given the mixed evidence, and the likely increasing importance of understanding how the public responds to news of progress against climate change, the topic deserves further attention. 

[^irena]: International Renewable Energy Agency, "Renewable Power Generation Costs in 2020" [(link to summary)](https://www.weforum.org/agenda/2021/07/renewables-cheapest-energy-source/). 

[^mckinsey]: Ole Rolser, Bram Smeets, and Rune van der Meijden, "Charting the global energy landscape to 2050: Emissions" [(link)](https://www.mckinsey.com/industries/oil-and-gas/our-insights/charting-the-global-energy-landscape-to-2050-emissions).

[^ipccreport]: IPCC, "AR6 Synthesis Report: Climate Change 2023", [(link)](https://www.ipcc.ch/report/sixth-assessment-report-cycle/).

We re-examine the effect of climate "good news" on public attitudes using an information provision experiment embedded in a pre-registered online survey of Americans ($n=2001$). Instead of simply exposing respondents to different messages and measuring their responses (a design that can exaggerate information effects and bias results toward researcher expectations), we use an "ask-tell" design [@braley2023voters]: after asking all respondents a series of factual questions about climate change mitigation, we randomly give some respondents the correct responses to select questions relating to the trajectory of US emissions, the cost of solar energy production, and the relative size of the renewable energy and coal sectors. Because most respondents underestimate progress in these areas, our information treatment conveys facts that most respondents would see as "good news". Finally, we survey respondents' views on climate change, their ability to address it, their support for policy responses, and their willingness to donate to a climate NGO. 

We find no effect of providing encouraging news about climate mitigation on any of our outcome measures, either on average or within subsets defined by pre-treatment pessimism on these topics, political ideology, party ID, and other characteristics. This suggests, contrary to some previous research, that broader appreciation of progress in climate mitigation will not erode support for further mitigation efforts, at least in the US. Because our information treatments appear not to have substantially changed respondents' perceptions of the tractability of climate mitigation, we view it as a still-open question how these perceptions affect citizens' support for further mitigation efforts. We can conclude, however, that while governments and NGOs should not expect to increase average support for climate action by advertising mitigation success stories, they should also not hesitate to do so out of concern that such messages will discourage action. 


## Theory

How might good (or bad) news about climate mitigation affect individuals' willingness to contribute to further mitigation efforts?  

A simple decision theoretic model helps to highlight some possibilities. Suppose an individual is deciding whether to make a costly contribution to a collective project (e.g. mitigation of GHG emissions) that may or may not succeed in the future. The individual receives a fixed utility benefit $B$ if the project succeeds, and contributing to the project costs $c$ (which could include financial costs as well as costs due to social pressure or cognitive dissonance). Whether this individual contributes then depends on $B$ and $c$ as well as the degree to which the individual's contribution increases the probability of the project's success (call this $\delta$). Specifically, she should contribute if $\delta B > c$. 

Now suppose the individual receives "good news" about the project --- i.e., information causing her to believe that others are contributing more than she thought, or that those contributions will be more effective than she thought. This information could affect her decision by changing $\delta$. Before receiving the good news, the individual may believe that the project is certain to fail with or without her contribution, so that $\delta B = 0 < c$; then the good news could make her think that her contribution could make a difference, so that $\delta B > c$ and she chooses to contribute. On the other hand, if the individual already was inclined to contribute, then good news could convince her that success is assured even without her contribution, reducing $\delta$ to zero and making it rational to not contribute.[^hornsey-link] 

[^hornsey-link]: Similarly, @hornsey2016cautionary note that good news about climate mitigation could motivate by making people feel more capable (their "motivational model") or demotivate by making people feel less concerned (their "complacency model").


Non-instrumental motives are likely important in decisions about contributing to a collective project like climate change mitigation. Suppose the individual's contribution has a negligible impact on the project's success ($\delta \approx 0$), so that a purely instrumental individual would never contribute. If the individual is a conformist who perceives the cost of contributing to be negative when enough others are contributing, or a team player who gets more benefit from success of the project when she herself has contributed, then information suggesting that many others are contributing and the project is likely to succeed could convince her to contribute. If she is a non-conformist or contrarian, it might have the opposite effect.  

Of course, good news about climate mitigation or another collective project could also have no impact on an individual's willingness to contribute. Beliefs about the likely success of the project or the effectiveness of individual action may be hard to change. The perceived costs of contributing $c$ could be high compared to the benefit of success $B$, so that small changes in $\delta$ don't matter. Moving further from the model, an individual could view climate mitigation as an ethical question or hold a fixed political position derived from political elites, such that her decision about contributing does not depend on her perception of the effectiveness of individual efforts. Whether individuals find "good news" about climate mitigation to be motivating, demotivating, or neither (on average or in subsets of a population) is therefore an empirical question to which we hope to contribute in this paper.

## Related literature

The most relevant previous paper is @hornsey2016cautionary, who study the effect of different messages about the trajectory of global CO$_2$ emissions on US respondents' emotions and motivation to take action against climate change. In their experiment, they present survey respondents with different framings of recent carbon emissions data: for example, the optimistic framing emphasizes that year-on-year global emissions were flat in 2014 for the first time in 40 years, while the pessimistic framing states that in 2014 "there was again no reduction in emission levels" (p.\ 30). @hornsey2016cautionary's findings suggest that good news about emissions may on average make people feel better but do less: respondents who see the optimistic message report more hope and less distress, but they also report less motivation to do something about climate change. 

Two recent studies show more modest impacts of emphasizing good or bad news about climate change mitigation efforts.  @clayton2020development ask US respondents to read an "empowering" message (e.g. "Wind and solar power are cheaper than ever and responsible for an increasing portion of energy use") or a "powerless" message (e.g. "Greenhouse gases contributing to global warming have long been on the rise and are now accelerating at their fastest pace in seven years"); most of their results are null, but they note that the empowering message reduced one of their climate anxiety measures (cognitive/emotional impairment). <!-- [^claytoncritique] --> @ettinger2021climate compare Americans' reactions to a "hope video" (which among other things emphasizes the declining cost of renewable energy) and a "doom and gloom video" (which among other things emphasizes that renewables would need to be expanded massively to meet climate targets); they find no effect on perceptions of climate risk, likelihood of behavior change, or likelihood of climate activism. 

Our analysis also relates to research examining how support for climate action depends on framing -- primarily, whether one emphasizes the risks of climate change or the economic and other benefits of successful mitigation.[^framinglit] Again, results are mixed. @lockwood2011does shows larger support for renewable energy among UK voters when it is framed as a path to energy independence rather than an economic opportunity or a way to tackle climate change; similarly, @bain2012promoting find that "framing climate change action as increasing consideration for others, or improving economic/technological development" was more motivating to Australian climate change skeptics than emphasizing the risks of climate change, and @dasandi2022positive detect a preference for emphasizing opportunities rather than threats in the US, UK, and China. [See also @dechezlepretre2022fighting, who compare videos about climate change impacts to videos about policy responses.] By contrast, @bernauer2016simple find no evidence that framing mitigation in terms of technological/economic benefits or community spirit increases support among American survey respondents (including skeptics) compared to a conventional frame of avoiding harms of climate change.

[^framinglit]: See @badullovich2020framing for a review of research on framing of climate change. @drews2016explains and @hornsey2020understanding review the broader research on public support for action on climate change.


### Advantages of our design {-} 

Our study contributes to this literature by using a pre-registered design to measure the impact of factual messages about the success of climate mitigation efforts on a large sample of Americans. Our design differs from previous approaches in important respects.

In typical experiments measuring the effect of information treatments or framing, respondents are asked to read a text [e.g. @hornsey2016cautionary; @clayton2020development] or watch a video [e.g. @ettinger2021climate; @dechezlepretre2022fighting] before answering questions on the subject of the text/video. Given the lack of pretext for presenting this information, survey respondents can  easily infer that the purpose of the survey is to test reactions to the information. If asked how they feel about climate change afterward, they may report feelings that align with the expected effect partly because they guess the researcher's objective and seek to align their responses with it [though see @mummolo2019demand]. (This demand effect may be exacerbated when, as in @hornsey2016cautionary, the researcher does not simply ask the respondent's attitude but instead asks the respondent how the stimulus affected their attitude.[^oops]) As a result, these studies may find that stimuli substantially affect attitudes even when the true effect is small or non-existent.

[^oops]: @dasandi2022positive similarly shows respondents two messages and asks respondents which message would affect their behavior more. @graham2021asking suggests that survey respondents are not reliable judges of how a treatment affects them.

We instead employ an "ask-tell" design [@ahler2014self; @braley2023voters], in which we selectively provide information in the context of a series of factual questions. This method of information provision has three main advantages compared to previous research in this area. First, because the preceding knowledge quiz asks respondents about the information we provide, we have a clear measure of what the respondent should learn from our information treatment, which is useful for interpreting treatment effects and can be used in subgroup analysis [@haaland2023designing].
Second, by offering a pretext for providing the information (i.e. that we wanted to provide answers to some of the questions on the quiz), we may reduce experimenter demand effects.  Third, participants may update factual beliefs more strongly when told that their existing knowledge is incorrect than when simply shown new information. 

Our design also advances on previous work by including a real-stakes outcome (a choice to donate money to a climate NGO or keep it). Such outcomes allow for a tougher test of information effects than typical attitudinal responses [@dechezlepretre2022fighting].   




## Research design


```{r}
library(tidyverse)
knitr::opts_chunk$set(dev = "ragg_png")
```





```{r load-survey}
qualtrics_load <- function(file_path){
  the_colnames <- read_csv(file_path) |> colnames()
  read_csv(file_path, skip = 3, col_names = F) |> 
    setNames(the_colnames) |>
    filter(consent == "Yes, I understand and consent to take part in the survey") |> 
    select(-StartDate, -EndDate, -Status, -Progress, -RecordedDate, -ResponseId, -DistributionChannel, -UserLanguage, -prolific_id, -consent)
}

raw_dat <- qualtrics_load("SSI3 final survey 2023 (SSD)_May 8, 2023_13.05.csv")
```

```{r process-survey}
great_deal_levels <- c("Not at all", "A little", "Somewhat", "A lot", "A great deal")
agreement_levels <- c("Strongly disagree", "Somewhat disagree", "Neither agree nor disagree", "Somewhat agree", "Strongly agree")

raw_dat |> 
  mutate(id = 1:n(), # for merging in knowledge scores later
         
         ### demographics
         education = factor(education, levels = c("Some high school", "High school graduate", "Some college", "2-year college degree", "4-year college degree", "Postgraduate degree (MA, MBA, MD, JD, PhD, etc.)"), ordered = T),
         lib_con = factor(lib_con, levels = c("Very conservative", "Conservative", "Moderate", "Liberal", "Very liberal"), ordered = T),
         hh_income = factor(hh_income, levels = c("Less than $25,000", "$25,000-$49,999", "$50,000-$74,999", "$75,000-$99,999", "$100,000-$149,999", "$150,000 or more", "Prefer not to say"), ordered = T),
         hh_income_missing = as.integer(hh_income == "Prefer not to say"),
         
         ### pre-treatment attitudes
         concern_cc_pre = factor(concern_cc_pre, levels = c("Not concerned at all", "Slightly concerned", "Moderately concerned", "Very concerned", "Extremely concerned"), ordered = T),
         efficacy_pre = factor(efficacy_pre, levels = agreement_levels, ordered = T),
         
         ### self-assessed knowledge beforehand 
         know_cc_pre = recode(know_cc_pre, !!!list("Not knowledgeable at all" = "Not at all", "Slightly knowledgeable" = "Slightly", "Moderately knowledgeable" = "Moderately", "Very knowledgeable" = "Very", "Extremely knowledgeable" = "Extremely")),
         know_cc_pre = factor(know_cc_pre, levels = c("Not at all", "Slightly", "Moderately", "Very", "Extremely"), ordered = T),
         
         # the info treatment knowledge responses
         know_jobs = factor(know_jobs, levels = c("Around 30,000, or one-fourth as many", "Around 60,000, or half as many", "Around 120,000, or the same number", "Around 240,000, or twice as many", "Around 480,000, or four times as many"), ordered = T),
         know_co2_change = factor(know_co2_change, levels = c("It went down by about 30%, to 4 billion tons", "It went down by about 15%, to 5 billion tons", "It stayed about the same (6 billion tons)", "It went up by about 15%, to 7 billion tons", "It went up by about 30%, to 8 billion tons"), ordered = T),
         know_solar_cost = factor(know_solar_cost, levels = c("The cost has gone down by around 80%", "The cost has gone down by about 40%", "The cost has stayed about the same", "The cost has gone up by about 40%", "The cost has gone up by about 80%"), ordered = T),

         ### post-treatment outcomes

         ## general attitudes/beliefs
         # concern about climate change
         how_worried = factor(how_worried, levels = c("Not at all worried", "A little worried", "Moderately worried", "Very worried", "Extremely worried"), ordered = T),
         how_affect_you = factor(how_affect_you, levels = great_deal_levels, ordered = T),
         how_affect_US = factor(how_affect_US, levels = great_deal_levels, ordered = T),
         how_affect_future = factor(how_affect_future, levels = great_deal_levels, ordered = T), 
         
         # efficacy measures
         believe_can_do_sthg = factor(believe_can_do_sthg, levels = agreement_levels, ordered = T),
         collective_efficacy = factor(collective_efficacy, levels = agreement_levels, ordered = T),

         # optimism
         emissions_prediction = factor(emissions_prediction, levels = c("Definitely not", "Probably not", "Possibly", "Probably", "Very probably", "Definitely"), ordered = T),
         
         # policy attitudes
         # attitude toward IRA
         ira_response = ifelse(is.na(ira_treatment_q), ira_control_q, ira_treatment_q),
         ira_response = factor(ira_response, levels = c("Goes much too far", "Goes somewhat too far", "Is about right", "Should go somewhat further", "Should go much further"), ordered = T),
         
         # willingness to pay taxes
         wtp_taxes = factor(wtp_taxes, levels = c("Totally unwilling", "A little willing", "Moderately willing", "Very willing", "Extremely willing"), ordered = T)) -> recoded_dat

recoded_dat |> 
  mutate(# efficacy index
         combined_efficacy_post = as.integer(believe_can_do_sthg) + as.integer(collective_efficacy), #  - 2)/8, # zero to 1
         
         # concern index
         concern_index = (as.integer(how_worried) + as.integer(how_affect_you) + as.integer(how_affect_US) + as.integer(how_affect_US)), #  - 4)/16, # so it's zero to 1

         # how many of the good news prompts did they get?
         # these are the information treatments
         n_good_news_wo_ira = as.integer(jobs_info == "included") + as.integer(us_emissions_info == "included") + as.integer(solar_cost_info == "included"), 
         # and this includes the IRA description as a piece of good news (which could affect concern etc)
         n_good_news_w_ira = n_good_news_wo_ira + as.integer(FL_39_DO == "FL_41|Concern,taxes,andefficacy"),
         
         # code correct knowledge questions
         correct_1_CO2 = as.integer(know_co2_meaning == "Carbon dioxide"),
         correct_2_GreenhouseEffect = as.integer(know_main_cause_gw == "Increased emissions of greenhouse gasses (the so-called greenhouse effect)"),
         correct_3_EarthquakesNotCC = as.integer(know_not_gw == "Earthquakes"),
         correct_4_WhatWouldHelp = as.integer(know_what_helps == "All of the above"),
         correct_5_WindIsRenewable = as.integer(know_renewable == "Wind"),
         correct_6_BidenPolicy = as.integer(know_ira_dcr == "Tax credits for people who purchase electric vehicles made in the US"),
         correct_7_USGasTaxLow = as.integer(know_gas_tax == 0),
         correct_8_SolarWindJobs = as.integer(know_jobs == "Around 480,000, or four times as many"),
         correct_9_USCO2Drop = as.integer(know_co2_change == "It went down by about 15%, to 5 billion tons"),
         correct_10_SolarCostDrop = as.integer(know_solar_cost == "The cost has gone down by around 80%"),
         
         ## NEW in revision: a measure of being at least as optimistic on each of the three good news pieces, and then a measure of how many "corrections" were received
         
         n_good_news_corrections = as.integer(jobs_info == "included")*(1 - correct_8_SolarWindJobs) + as.integer(us_emissions_info == "included")*(1 - as.integer(know_co2_change %in% c("It went down by about 15%, to 5 billion tons", "It went down by about 30%, to 4 billion tons"))) + as.integer(solar_cost_info == "included")*(1 - correct_10_SolarCostDrop),
         number_possible_corrections = (1 - correct_8_SolarWindJobs) + (1 - as.integer(know_co2_change %in% c("It went down by about 15%, to 5 billion tons", "It went down by about 30%, to 4 billion tons"))) + (1 - correct_10_SolarCostDrop)
         ) -> recoded_and_augmented_dat

# get pct correct 
recoded_and_augmented_dat |> 
  select(id, starts_with("correct_")) |> 
  pivot_longer(cols = -id, names_prefix = "correct_", names_to = "question", values_to = "correct") |> 
  group_by(id) |> 
  summarize(pct_correct = mean(correct, na.rm = T)) -> correct_pcts

recoded_and_augmented_dat |> 
  select(id, starts_with("correct_")) |> 
  pivot_longer(cols = -id, names_prefix = "correct_", names_to = "question", values_to = "correct") |> 
  filter(str_detect(question, "^8") | str_detect(question, "^9") | str_detect(question, "^10")) |> 
  group_by(id) |> 
  summarize(pct_correct_good_news = mean(correct, na.rm = T)) -> correct_pcts_gn

recoded_and_augmented_dat |> 
  left_join(correct_pcts, by = "id") |> 
  left_join(correct_pcts_gn, by = "id") -> dat_a 

dat_a |> 
  mutate(pessimism_jobs = -1*as.integer(know_jobs) + 5,
         pessimism_co2_change = -1*as.integer(know_co2_change) + 5,
         pessimism_solar = as.integer(know_co2_change) - 1,
         pessimism_index = (pessimism_jobs + pessimism_co2_change + pessimism_solar)/12,
         pessimism_tercile = factor(ntile(pessimism_index, n = 3), levels = c("1", "2", "3"), ordered = T),
         age_group = case_when(age < 35 ~ "18-34",
                               age < 50 ~ "35-49",
                               TRUE ~ "50+"),
         age_group = factor(age_group, levels = c("18-34", "35-49", "50+")),
         ideology = case_when(lib_con < "Moderate" ~ "Conservative",
                              lib_con < "Liberal" ~ "Moderate",
                              lib_con > "Moderate" ~ "Liberal"),
         ideology = factor(ideology, levels = c("Conservative", "Moderate", "Liberal")),
         party_id = ifelse(party_id == "Other", NA, party_id),
         party_id = factor(party_id, levels = c("Republican", "Independent", "Democrat")),
         concern_group = case_when(concern_cc_pre < "Moderately concerned" ~ "Not concerned or slightly concerned",
                                   concern_cc_pre == "Moderately concerned" ~ "Moderately concerned",
                                   concern_cc_pre > "Moderately concerned" ~ "Very or extremely concerned"),
         concern_group = factor(concern_group, levels = c("Not concerned or slightly concerned", "Moderately concerned", "Very or extremely concerned")),
         efficacy_group = case_when(efficacy_pre < "Somewhat agree" ~ "Low efficacy",
                                    efficacy_pre == "Somewhat agree" ~ "Medium efficacy",
                                    TRUE ~ "High efficacy"),
         efficacy_group = factor(efficacy_group, levels = c("Low efficacy", "Medium efficacy", "High efficacy"))) |> 
  # get standardized DVs 
  mutate(efficacy_standardized = combined_efficacy_post/sd(combined_efficacy_post, na.rm = T),
         emissions_prediction_standardized = as.integer(emissions_prediction)/sd(as.integer(emissions_prediction, na.rm = T)),
         concern_index_standardized = concern_index/sd(concern_index, na.rm = T),
         wtp_taxes_standardized = as.integer(wtp_taxes)/sd(as.integer(wtp_taxes), na.rm = T),
         ira_response_standardized = as.integer(ira_response)/sd(as.integer(ira_response), na.rm = T),
         donate_amount_standardized = donate_amount_4/sd(donate_amount_4, na.rm = T)) |> 
  # recenter covariates
  mutate(concern_cc_pre_raw = concern_cc_pre,
         efficacy_pre_raw = efficacy_pre,
         hh_income_raw = hh_income,
         concern_cc_pre = as.integer(concern_cc_pre) - mean(as.integer(concern_cc_pre), na.rm = T),
         efficacy_pre = as.integer(efficacy_pre) - mean(as.integer(efficacy_pre), na.rm = T),
         hh_income = as.integer(hh_income) - mean(as.integer(hh_income), na.rm = T)) -> dat

```



### Sample characteristics

A sample of 2001 Prolific participants completed our survey on May 7, 2023. Some characteristics of the sample appear in @tbl-sum-stats. The sample was balanced by design on gender only (49.6% female). Our sample is otherwise fairly representative of the US population in terms of race (77% white vs 75.5% in the US Census's 2022 population estimates,[^popest] 9.3% Black or African-American vs 13.6%, 7.3% Asian vs 6.3%, 8.7% Hispanic vs 19.1%) and age (median age of 38 vs 38.8 in the census, though about 50 in the electorate[^medianage]). As is often the case in online surveys, our sample is more educated than the population (only 14.6% did not attend college vs 36% in the census) and includes fewer households earning above $100k (22.1% vs 36%); our sample is also strongly disproportionately left-leaning (50.4% identify as Democrats, 16.7% as Republicans, 29.3% as Independents vs 29%, 30%, and 38% in a contemporaneous Gallup poll[^gallup]). Compared to the US electorate, our sample thus over-represents the kind of young, left-leaning voter who tends to be concerned about climate change and favors climate action [@bumann2021determinants], and our results are especially informative about how such voters may react to news of progress in climate mitigation. 

```{r}
#| label: tbl-sum-stats
#| tbl-cap: 'Characteristics of the sample'
#| 
# we have four concern questions, efficacy, 
tribble(~Characteristic, ~Proportion,
        "Female", mean(dat$sex == "Female", na.rm = T),
        "White", mean(dat$race == "White", na.rm = T), #sd(dat$race == "White", na.rm = T),
        "Black or African American", mean(dat$race == "Black or African American", na.rm = T), #sd(dat$race == "Black or African American", na.rm = T),
        "Asian", mean(dat$race == "Asian", na.rm = T), #sd(dat$race == "Asian", na.rm = T),
        "Hispanic", mean(dat$hispanic == "Yes", na.rm = T),
        "Age <= 30", mean(dat$age <= 30, na.rm = T), #sd(dat$age, na.rm = T),
        "Age 31-50", mean(dat$age > 30 & dat$age <= 50, na.rm = T),
        "Age 70+", mean(dat$age >= 70, na.rm = T),
        "No more than HS education", mean(dat$education <= "High school graduate", na.rm = T), #sd(dat$education <= "High school graduate", na.rm = T),
        "4-year college degree or more", mean(dat$education >= "4-year college degree", na.rm = T), #sd(dat$education >= "4-year college degree", na.rm = T),
        "Household income < \\$50k", mean(dat$hh_income_raw < "$50,000-$74,999", na.rm = T),
        "Household income > \\$100k", mean(dat$hh_income_raw >= "$100,000-$149,999", na.rm = T),
        "Democrat", mean(dat$party_id == "Democrat", na.rm = T), #sd(dat$party_id == "Democrat", na.rm = T),
        "Independent", mean(dat$party_id == "Independent", na.rm = T), # sd(dat$party_id == "Independent", na.rm = T),
        "Republican", mean(dat$party_id == "Republican", na.rm = T), # sd(dat$party_id == "Republican", na.rm = T),
        "Very concerned about climate change (pre-treatment)", mean(as.character(dat$concern_cc_pre_raw) == "Very concerned", na.rm = T), # sd(dat$concern_cc_pre_raw == "Very concerned", na.rm = T),
        "Extremely concerned about climate change (pre-treatment)", mean(dat$concern_cc_pre_raw == "Extremely concerned", na.rm = T), #sd(dat$concern_cc_pre_raw == "Extremely concerned", na.rm = T),
        "Strongly agree w. `I believe I can do something to address climate change' (pre-treatment)", mean(dat$efficacy_pre_raw == "Strongly agree", na.rm = T) #, sd(dat$efficacy_pre_raw == "Strongly agree", na.rm = T)
        ) -> full_table 

full_table |> slice(1:9) |> mutate(Proportion = round(Proportion, 2)) |> 
  mutate(`Characteristic ` = full_table |> slice(10:18) |> pull(Characteristic),
         `Proportion ` = full_table |> slice(10:18) |> pull(Proportion) |> round(2)) |> 
  kableExtra::kbl(booktabs = T, escape = F, linesep = "") |> 
  kableExtra::column_spec(1, width = "15em") |> 
  kableExtra::column_spec(2, width = "5em") |> 
  kableExtra::column_spec(3, width = "20em") |>
  kableExtra::column_spec(4, width = "5em") |>
  kableExtra::kable_styling(font_size = 11, latex_options = c("scale_down", "striped"))  #  "scale_down", 
```


[^popest]: US Census Bureau, "QuickFacts" [(link)](https://www.census.gov/quickfacts/fact/table/US/PST045222), visited Aug 23, 2023.

[^medianage]: Pew Research Center, "What the 2020 electorate looks like by party, race and ethnicity, age, education and religion" [(link)](https://www.pewresearch.org/short-reads/2020/10/26/what-the-2020-electorate-looks-like-by-party-race-and-ethnicity-age-education-and-religion/), visited Aug 23, 2023.

[^gallup]: Gallup, "Party Affiliation" [(link)](https://news.gallup.com/poll/15370/party-affiliation.aspx), visited Aug 23, 2023.

We included an attention check asking respondents to select two specific responses from a five-point agree/disagree Likert scale. (Exact wording appears in the appendix.)  Only 14 respondents (.7%) selected just one response, indicating that they ignored the instructions entirely; another 92 respondents (4.6%) selected the wrong pair of responses, indicating some inattentiveness. Below we retain these respondents to simplify interpretation. In the Appendix we reproduce @fig-coef-plot and the average effects regression table without these 106 respondents, producing very similar conclusions.


<!-- We fielded a survey to 2,000 US residents on May 7, 2023 via Prolific.  --> 

### Structure of the survey 

After collecting background characteristics, we ask respondents to answer ten factual questions about climate change as part of a "knowledge quiz". (All questions and answers appear in Appendix @tbl-quiz-questions-appx.) We ask respondents not to look up the answers and inform them that we will give them the correct answers later in the survey.  The final three questions ask about facts that later form the basis of our "good news" information treatments. These three questions are listed (along with the correct answers) in @tbl-quiz-questions. 
We designed these questions so that informing respondents of the answer might make them more optimistic about climate change mitigation (particularly in the US); indeed, <!-- in a pilot study of 100 respondents, --> 73% gave a response that was more pessimistic than the correct one about changes in US emissions, 92% underestimated the drop in the cost of solar power, and 96% underestimated the size of the solar and wind sector relative to coal.[^goodnewscaveat] 

[^goodnewscaveat]: Our information treatments are not "good news" in any more general sense. For example, the relative size of the renewables sector vs. coal could be bad news for someone with a personal stake in the fossil fuel industry.  

```{r summarize-pess-survey, eval = F}
# this is where those numbers in the footnote come from 
dat |> 
  mutate(pess_jobs = know_jobs != "Around 480,000, or four times as many",
         pess_solar = know_solar_cost != "The cost has gone down by around 80%", na.rm = T,
         pess_us_emissions =!know_co2_change %in% c("It went down by about 15%, to 5 billion tons", "It went down by about 30%, to 4 billion tons"),
         pess_something = pess_jobs | pess_solar | pess_us_emissions) |> 
  summarize(across(c(pess_jobs, pess_solar, pess_us_emissions, pess_something), mean, na.rm = T))
```

```{r tbl-quiz-questions}
#| label: tbl-quiz-questions
#| tbl-cap: Questions on our knowledge survey that form the basis of our "good news" information treatments

tribble(~Question, ~`Correct answer`, ~`Other answer options`,
        "What does CO2 stand for?", "Carbon dioxide", "Carbon monoxide; Greenhouse effect; Climate change",
        "Which of the following phenomena is the main cause of global warming over the last 20 years?", "Increased emissions of greenhouse gasses (the so-called greenhouse effect)", "Reduction of the ozone layer (the so-called ozone hole); Changes in ocean currents, e.g., 'el Niño'; Changes in the tilt of earth’s axis",
        "Which phenomenon is not caused by global warming?", "Earthquakes", "Glacial melting; Sea level rise; Changes in ocean currents",
        "Which of the following do scientists believe would help limit global warming?", "All of the above", "Making buildings more energy efficient, Increasing the use of electric vehicles, Generating more electricity from solar energy, Reducing the consumption of meat",
        "Which of the following is a form of renewable energy?", "Wind", "Clean coal; Petroleum; Natural gas",
        "Which of the following is part of environmental legislation passed by the Biden administration?", "Tax credits for people who purchase electric vehicles made in the US", "New taxes on the consumption of gasoline and other fossil fuels; A nationwide cap-and-trade system to limit carbon emissions; Federal funding for the construction of large hydropower dams",
        "Taxes on gasoline are found in all OECD countries. (The OECD is an organization of relatively wealthy countries such as the US, Canada, France, Germany, Australia and Japan.) These taxes are considered an important way to convince people to drive less and use more efficient vehicles. How many of the 37 other OECD countries do you think have lower gasoline taxes than the US?", "0", "14; 26, 27",
        "About 120,000 Americans currently work in the coal industry. How many Americans work in the solar and wind power industries?", "Around 480,000, or four times as many", "Around 30,000, or one-fourth as many; Around 60,000, or half as many; Around 120,000, or the same number; Around 240,000, or twice as many",
        "In 2007, the US emitted about 6 billion tons of carbon dioxide from fossil fuels. How has that number changed since then?", "It went down by about 15%, to 5 billion tons", "It went down by about 30%, to 4 billion tons; It stayed about the same (6 billion tons); It went up by about 15%, to 7 billion tons; It went up by about 30%, to 8 billion tons",
        "In the US and elsewhere, large solar power plants are built to convert sunlight into electricity for the grid. How has the cost of building such projects changed over the last 10 years?", "The cost has gone down by around 80%","The cost has gone down by about 40%; The cost has stayed about the same; The cost has gone up by about 40%; The cost has gone up by about 80%") -> knowledge_quiz

knowledge_quiz |> 
  slice(8:10) -> info_treatments 

info_treatments |> 
  mutate(Shorthand = c("Renewables jobs", "US emissions", "Solar cost"),
         `Contextual statement (provided only to treated respondents)` = c("Building a solar plant has become the cheapest-ever way to generate electricity. Further investment in transmission and storage are necessary to make the best use of cheap and clean power from solar and other renewable sources.", "Increasing use of renewable energy is expected to cause further job growth in this sector. Analysts believe that political support from the fast-growing clean energy sector could lead to more proactive climate policy in the future.", "This drop has occurred mainly because we are using less coal, more natural gas, and more renewables. A similar drop has taken place in Europe. Experts say that we can cut a lot more by generating more electricity from low-carbon sources and plugging more of our machinery (cars, industrial boilers, etc) into the electrical grid.")) |> 
  relocate(Shorthand) |> 
  kableExtra::kbl(booktabs = T, linesep = "") |> 
  kableExtra::column_spec(1, width = "5em") |>   # 17
  kableExtra::column_spec(2, width = "9em") |>   # 17
  kableExtra::column_spec(3, width = "4em") |>  # 10
  kableExtra::column_spec(4, width = "10em") |>    # 17
  kableExtra::column_spec(5, width = "18em") |>
  kableExtra::kable_styling(font_size = 9, latex_options = c("striped")) # |>   # , "scale_down"
  # kableExtra::landscape()                             

```

Following the knowledge quiz we randomly provide answers to these three "good news" questions. Specifically, after the respondent has answered all questions on the knowledge quiz, we randomly determine whether the respondent is to be shown the correct answer to each of these questions before proceeding to the rest of the survey. These randomizations are independent, so that respondents are assigned to see anywhere between 0 and 3 correct answers. Those who are not assigned to receive any information treatments proceed directly to the next section. Those who are assigned to receive at least one correct answer are told that before moving to the next section of the survey "we want to share with you a few facts from our knowledge quiz that many participants find surprising," after which we provide the information treatments for which the respondent was randomly selected. In each of these information treatments, we remind the respondent about the question that was asked, we remind them of the answer they provided, and we tell them if they were correct or incorrect. If they were incorrect we tell them the correct answer. We also provide a source for the correct answer and a short contextual statement (appearing in the last column of @tbl-quiz-questions) that links the fact to the US's efforts to reduce GHG emissions. Because the three information treatments are administered according to a factorial design, we can measure the effect of each of these pieces of information separately (averaging over the other treatment statuses) and also measure the effect of providing more vs.\ less information; we do both below.

Next, we ask all respondents a set of questions about respondents' attitudes toward climate change.  

- **Beliefs and emotions**: We ask a question about the respondent's confidence that global CO$_2$ emissions will be cut in half by 2050, four questions about the respondent's concern about climate change from @lawson2019children (the answers to which we sum to create a concern index), and two questions about the respondent's sense of efficacy adapted from @clayton2020development (the answers to which we sum to create an efficacy index). These questions are designed to capture beliefs and emotions that could inform willingness to contribute to climate mitigation. 
- **Willingness to contribute to climate mitigation**: We measure the respondent's willingness to pay taxes to support climate change policy, their view on whether the Inflation Reduction Act (IRA) goes too far or should go further, and (after explaining that they have been entered into a lottery for a $100 prize) their willingness to donate some or all of their winnings to Evergreen Action, a policy advocacy group that contributed to the development of the IRA and is working on IRA implementation.

Details on the wording and response options, as well as the mean and standard deviation of each outcome variable, are provided in @tbl-outcome-vars. In all analysis below we use standardized versions of each outcome (i.e. the raw measure divided by its standard deviation). 

```{r}
#| label: tbl-outcome-vars
#| tbl-cap: 'Outcome variables in the survey'

tibble(`Outcome variable` = c("Optimism about future emissions", "Concern about climate change", "Efficacy index", "Willingness to pay higher taxes", "Support for aggressive policy", "Donation"),
       `Question wording` = c("'The Intergovernmental Panel on Climate Change has stated that to limit global warming to 2 degrees Celsius above pre-industrial levels, global CO2 emissions will need to drop by more than half by 2050. How likely do you think it is that we will cut emissions by at least that amount?'", "(1) 'How worried are you about climate change?', 'How much do you think climate change will' (2) 'affect you personally?', (3) 'negatively affect people in the United States?', (4) 'negatively affect future generations of people?'", "'I believe I can do something to help address the problem of climate change.' and 'I believe that by working together we can reduce greenhouse gas emissions and address climate change.'", "'Would you be willing to pay higher taxes to support a more aggressive national effort to fight climate change?'", "[Following description of IRA:] 'Do you think the IRA goes too far or not far enough in addressing climate change?'", "'By taking this survey, you are automatically entered into a lottery to win \\$100. [Description of Evergreen Action] Please use the slider below to indicate how much of your lottery winnings (between \\$0 and \\$100) you would like us to donate to Evergreen Action, should you win.'"),
       `Response options` = c("From `Definitely not' (1) to `Definitely' (6)", "Sum of responses on four questions, from `Not at all' (1) to `A great deal' (5)", "Sum of responses on two questions, from `Strongly disagree' (1) to `Strongly agree' (5)", "From `Totally unwilling' (1) to `Extremely willing' (5)", "From `Goes much too far' (1) to `Should go much further' (5)", "Integer values from 0 to 100"),
       `Mean` = round(c(mean(as.integer(dat$emissions_prediction, na.rm = T)), mean(dat$concern_index, na.rm = T), mean(dat$combined_efficacy_post, na.rm = T), mean(as.integer(dat$wtp_taxes) , na.rm = T), mean(as.integer(dat$ira_response), na.rm = T), mean(dat$donate_amount_4, na.rm = T)), 2),
       `Std. Dev.` = round(c(sd(as.integer(dat$emissions_prediction, na.rm = T)), sd(dat$concern_index, na.rm = T), sd(dat$combined_efficacy_post, na.rm = T), sd(as.integer(dat$wtp_taxes) , na.rm = T), sd(as.integer(dat$ira_response), na.rm = T), sd(dat$donate_amount_4, na.rm = T)), 2)) |> 
  kableExtra::kbl(booktabs = T, escape = F) |> 
  kableExtra::column_spec(1, width = "4em") |> 
  kableExtra::column_spec(2, width = "30em") |> 
  kableExtra::column_spec(3, width = "8em") |> 
  kableExtra::column_spec(4, width = "2.5em") |>
  kableExtra::column_spec(5, width = "2.5em") |>
  kableExtra::kable_styling(font_size = 9, latex_options = "striped")

```


At the very end of the survey, all respondents are shown correct responses to all questions on the knowledge quiz. We randomly chose one respondent as the lottery winner and (following that respondent's instructions) we awarded \$50 to that respondent and \$50 to Evergreen Action.  

## Results 

<!-- multiple hypothesis testing corrections for analysis of average effects --> 

```{r mht-avg, cache = T}
# plan: reshuffle treatment, record key t-stats for each outcome
# determine single-test t-stat cutoff such that we have at least one false positive per family no more than .05 of the time

stats_for_dat <- function(dat){
  
  outcomes <- c("emissions_prediction_standardized", "concern_index_standardized", "efficacy_standardized",
                "wtp_taxes_standardized", "ira_response_standardized", "donate_amount_standardized")
  outcome_labels <- c("DV: Optimism about\nfuture emissions", "DV: Concern about\nclimate change", "DV: Efficacy index",
                      "DV: Willingness to\npay higher taxes", "DV: Support for\naggressive policy", "DV: Donation")
  controls <- c("as.integer(concern_cc_pre) + as.integer(lib_con)", 
                "as.integer(concern_cc_pre) + as.integer(lib_con)",
                "as.integer(efficacy_pre) + as.integer(lib_con)", 
                "as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing",
                "as.integer(concern_cc_pre) + as.integer(lib_con)",
                "as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing")

  tibble(outcome = outcomes,
       outcome_label = outcome_labels,
       controls = controls) |> 
  mutate(the_formula = map(str_c(outcome, " ~ n_good_news_wo_ira + ", controls), as.formula),
         reg = map(the_formula, ~ estimatr::lm_robust(formula = .x, data = dat)),
                   tidied = map(reg, broom::tidy)) |> 
  select(-outcome) |> 
  unnest(tidied) |> 
  filter(term == "n_good_news_wo_ira") |> 
  select(outcome_label, p.value, statistic)
}

set.seed(12345)
tibble(j = 1:1000) |> 
  # reshuffle treatment
  mutate(this_dat = map(j, ~ mutate(dat, n_good_news_wo_ira = sample(n_good_news_wo_ira))),
         sfd = map(this_dat, stats_for_dat)) |>
  select(-this_dat) |> 
  unnest(sfd) |> 
  mutate(family = ifelse(outcome_label %in% c("DV: Optimism about\nfuture emissions", "DV: Concern about\nclimate change", "DV: Efficacy index"), "A", "B")) -> longer

reject_or_nah <- function(data, p_star){
  sum(data$p.value < p_star) > 0
}

# choose a t-stat or p-value cutoff. count how many family-wise rejections we get 
longer |> 
  expand_grid(p_star = seq(.01, .03, length = 100)) |> 
  arrange(p_star, j) |> 
  group_by(j, family, p_star) |> 
  nest() |> 
  mutate(reject = map2_lgl(data, p_star, reject_or_nah)) |> 
  group_by(p_star, family) |> 
  summarize(reject_rate = mean(reject)) -> summarized 

family_A_cutoff <- summarized |> 
  ungroup() |> 
  filter(family == "A" & reject_rate < .05) |> 
  arrange(desc(p_star)) |> 
  slice(1) |> 
  pull(p_star)

family_B_cutoff <- summarized |> 
  ungroup() |> 
  filter(family == "B" & reject_rate < .05) |> 
  arrange(desc(p_star)) |> 
  slice(1) |> 
  pull(p_star)

# .0136 for A, .0151 for B. 
```


```{r mht-itx, cache = T}
# in PAP I said we would check the intx between n_good_news_wo_ira and highest pessimism tercile.
# but I think I should not have done that. I think it should have been checking all interactions -- guarding against type 1 error in identifying a significant interaction.

stats_for_dat_intx <- function(dat){
  
  outcomes <- c("emissions_prediction_standardized", "concern_index_standardized", "efficacy_standardized",
                "wtp_taxes_standardized", "ira_response_standardized", "donate_amount_standardized")
  outcome_labels <- c("DV: Optimism about\nfuture emissions", "DV: Concern about\nclimate change", "DV: Efficacy index",
                      "DV: Willingness to\npay higher taxes", "DV: Support for\naggressive policy", "DV: Donation")
  controls <- c("as.integer(concern_cc_pre) + as.integer(lib_con)", 
                "as.integer(concern_cc_pre) + as.integer(lib_con)",
                "as.integer(efficacy_pre) + as.integer(lib_con)", 
                "as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing",
                "as.integer(concern_cc_pre) + as.integer(lib_con)",
                "as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing")

  tibble(outcome = outcomes,
       outcome_label = outcome_labels,
       controls = controls) |> 
  mutate(the_formula = map(str_c(outcome, " ~ n_good_news_wo_ira*pessimism_tercile + ", controls), as.formula),
         reg = map(the_formula, ~ estimatr::lm_robust(formula = .x, data = dat)),
                   tidied = map(reg, broom::tidy)) |> 
  select(-outcome) |> 
  unnest(tidied)  |>
  filter(str_detect(term,  "n_good_news_wo_ira:pessimism_tercile")) |>   # was .Q
  select(outcome_label, p.value, term, statistic)
}

set.seed(12345)
tibble(j = 1:1000) |> 
  # reshuffle treatment
  mutate(this_dat = map(j, ~ mutate(dat, n_good_news_wo_ira = sample(n_good_news_wo_ira))),
         sfd = map(this_dat, stats_for_dat_intx)) |>
  select(-this_dat) |> 
  unnest(sfd) |> 
  mutate(family = ifelse(outcome_label %in% c("DV: Optimism about\nfuture emissions", "DV: Concern about\nclimate change", "DV: Efficacy index"), "A", "B")) -> longer

# choose a t-stat or p-value cutoff. count how many family-wise rejections we get 
longer |> 
  expand_grid(p_star = seq(.001, .025, length = 100)) |> 
  arrange(p_star, j) |> 
  group_by(j, family, p_star) |> 
  nest() |> 
  mutate(reject = map2_lgl(data, p_star, reject_or_nah)) |> 
  group_by(p_star, family) |> 
  summarize(reject_rate = mean(reject)) -> summarized_itx 

family_A_cutoff_itx <- summarized_itx |> 
  ungroup() |> 
  filter(family == "A" & reject_rate < .05) |> 
  arrange(desc(p_star)) |> 
  slice(1) |> 
  pull(p_star)

family_B_cutoff_itx <- summarized_itx |> 
  ungroup() |> 
  filter(family == "B" & reject_rate < .05) |> 
  arrange(desc(p_star)) |> 
  slice(1) |> 
  pull(p_star)

```


```{r coef-plot-prep, warning = F, echo = F}
# a function to get confidence intervals for subsets from the interactive models
tercile_gaps <- function(mod,
                         group_names = c("No children", "One or two children", "Three or more children"),
                         terms = NULL){
  if(is.null(terms)){coef_names <- names(coef(mod))[!names(coef(mod)) %in% c("(Intercept)")]; terms <- coef_names[c(1, length(coef_names) - 1, length(coef_names))]}
  tibble(group = group_names,
         estimate = coef(mod)[terms[1]] + 
           c(0, 
             mod |> broom::tidy() |> filter(term == terms[2]) |> pull(estimate), 
             mod |> broom::tidy() |> filter(term == terms[3]) |> pull(estimate)),
         std.error = sqrt(vcov(mod)[terms[1], terms[1]] + 
                            c(0,
                              # the 2 was missing in the original submission
                              vcov(mod)[terms[2], terms[2]] + 2*vcov(mod)[terms[1], terms[2]],
                              vcov(mod)[terms[3], terms[3]] + 2*vcov(mod)[terms[1], terms[3]]))) |> 
    mutate(ci_lower = estimate - 1.96*std.error,
           ci_upper = estimate + 1.96*std.error)
}  

outcomes <- c("emissions_prediction_standardized", "concern_index_standardized", "efficacy_standardized", "wtp_taxes_standardized", "ira_response_standardized", "donate_amount_standardized")
outcome_labels <- c("DV: Optimism about\nfuture emissions", "DV: Concern about\nclimate change", "DV: Efficacy index", "DV: Willingness to\npay higher taxes", "DV: Support for\naggressive policy", "DV: Donation")
interaction_variables <- c("age_group", "ideology", "party_id", "pessimism_tercile", "concern_group", "efficacy_group")
interaction_labels <- c("By age (low=18-34,\nmed=35-49,high=50+)", "By liberal-ness (low=con,\nmed=mod,high=lib)", "By Democrat-ness (low=R,\nmed=I,high=D)", "By pre-treatment\nfactual pessimism", "By pre-treatment concern", "By pre-treatment efficacy")
controls <- c("as.integer(concern_cc_pre) + as.integer(lib_con)", "as.integer(concern_cc_pre) + as.integer(lib_con)", "as.integer(efficacy_pre) + as.integer(lib_con)", "as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing", "as.integer(concern_cc_pre) + as.integer(lib_con)","as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing")

# regressions for average effects
tibble(outcome = outcomes,
       outcome_label = outcome_labels,
       controls = controls) |> 
  mutate(the_formula = map(str_c(outcome, " ~ n_good_news_wo_ira + ", controls), as.formula),
         reg = map(the_formula, ~ estimatr::lm_robust(formula = .x, data = dat))) -> ate_regs 

ate_regs |> 
  mutate(tidied = map(reg, broom::tidy),
         n = map_int(reg, nobs)) |> 
  select(-outcome) |> 
  unnest(tidied) |> 
  mutate(group = "All",
         interaction_label = "All") |> 
  filter(term == "n_good_news_wo_ira") |> 
  select(estimate, ci_lower = conf.low, ci_upper = conf.high, interaction_label, group, outcome_label, n, p.value) -> ate_regs2

# regressions for the interaction models
expand_grid(tibble(interaction_variable = interaction_variables,
                   interaction_label = interaction_labels,
                   group_names = list(c("Low", "Medium", "High"))),
            tibble(outcome = outcomes,
                   outcome_label = outcome_labels,
                   controls = controls)) |> 
  mutate(the_formula = map(str_c(outcome, " ~ n_good_news_wo_ira *", interaction_variable, " + ", controls), as.formula),
         reg = map(the_formula, ~ estimatr::lm_robust(formula = .x, data = dat))) -> ate_intx_regs 


ate_intx_regs |> 
  mutate(tg = map2(reg, group_names, tercile_gaps),
         n = map_int(reg, nobs)) |> 
  unnest(tg) |> 
  select(estimate, ci_lower, ci_upper, interaction_label, group, outcome_label, n) -> atx_regs2 

key_intx_reg <- ate_intx_regs |> 
  filter(interaction_variable == "pessimism_tercile" & outcome == "donate_amount_standardized") |> 
  pull(reg)

key_p_value <- broom::tidy(key_intx_reg[[1]]) |> 
  filter(term == "n_good_news_wo_ira:pessimism_tercile.Q") |> 
  pull(p.value)
  
bind_rows(ate_regs2, atx_regs2) |> 
  mutate(group = fct_inorder(group),
         outcome_label = fct_inorder(outcome_label), 
         interaction_label = fct_inorder(interaction_label)) -> for_figure_1

```

Following our pre-analysis plan (PAP),[^pap-dev] we estimated the effect of providing good news on each standardized outcome using linear regression with robust standard errors. For each outcome, we regress the variable on the number of correct answers we provided the respondent (0-3)[^correct-answers] plus the (recentered) covariates we found to be most predictive of that outcome in the pilot;[^covariates-specific] in the figures below we report the coefficient and 95% confidence interval for the estimated effect of an additional piece of good news. To study treatment effect heterogeneity, we add an interaction between subgroup membership indicators (e.g. levels of pre-treatment concern about climate change) and the number of pieces of good news provided; we derive the estimated effect for each subgroup from this regression and report these effects with 95% confidence intervals in the figures. Our PAP stated that we would study heterogeneity by pre-treatment factual pessimism,[^pessimism] but we show exploratory results for other interactions. The appendix contains the regression tables underlying all estimates. 

[^pap-dev]: Our PAP did not state that we would standardize outcomes. This and three other deviations are stated in the paper and explained further in the appendix.

[^correct-answers]: Two anonymous reviewers suggested that we take into account respondents' answers on the knowledge quiz in defining the treatment. In additional analysis reported in Appendix @fig-coef-plot-a, we use as the treatment the number of times we gave the respondent the correct answer to a question to which the respondent had given an overly-pessimistic answer. The results are almost identical to those using the number of correct answers we provide. <!-- new -->    

[^covariates-specific]: These were: political ideology for all outcomes; pre-treatment efficacy for the "Efficacy" outcome and pre-treatment climate concern for the other five outcomes; and household income for "Willingness to pay higher taxes" and "Donation."  

[^pessimism]: We coded respondents' answers to each of the three "good news" questions on the knowledge survey on a numerical scale of pessimism (0 points for saying that the cost of solar had dropped by 80%, 1 point for saying it had dropped by 40%, etc). We sum these scores across the three questions to get a pessimism score and divide respondents into terciles.


```{r}
# regressions for average effects with the other treatment variable
tibble(outcome = outcomes,
       outcome_label = outcome_labels,
       controls = controls) |> 
  mutate(the_formula = map(str_c(outcome, " ~ n_good_news_corrections + ", controls), as.formula),
         reg = map(the_formula, ~ estimatr::lm_robust(formula = .x, data = dat, fixed_effects = number_possible_corrections))) -> ate_regs_a 

ate_regs_a |> 
  mutate(tidied = map(reg, broom::tidy),
         n = map_int(reg, nobs)) |> 
  select(-outcome) |> 
  unnest(tidied) |> 
  mutate(group = "All",
         interaction_label = "All") |> 
  filter(term == "n_good_news_corrections") |> 
  select(estimate, ci_lower = conf.low, ci_upper = conf.high, interaction_label, group, outcome_label, n, p.value) -> ate_regs2_a

# regressions for the interaction models with the other treatment variable
expand_grid(tibble(interaction_variable = interaction_variables,
                   interaction_label = interaction_labels,
                   group_names = list(c("Low", "Medium", "High"))),
            tibble(outcome = outcomes,
                   outcome_label = outcome_labels,
                   controls = controls)) |> 
  mutate(the_formula = map(str_c(outcome, " ~ n_good_news_corrections *", interaction_variable, " + ", controls), as.formula),
         reg = map(the_formula, ~ estimatr::lm_robust(formula = .x, data = dat, fixed_effects = number_possible_corrections))) -> ate_intx_regs_a 


ate_intx_regs_a |> 
  mutate(tg = map2(reg, group_names, tercile_gaps),
         n = map_int(reg, nobs)) |> 
  unnest(tg) |> 
  select(estimate, ci_lower, ci_upper, interaction_label, group, outcome_label, n) -> atx_regs2_a 

key_intx_reg_a <- ate_intx_regs_a |> 
  filter(interaction_variable == "pessimism_tercile" & outcome == "donate_amount_standardized") |> 
  pull(reg)

key_p_value_a <- broom::tidy(key_intx_reg_a[[1]]) |> 
  filter(term == "n_good_news_wo_ira:pessimism_tercile.Q") |> 
  pull(p.value)
  
bind_rows(ate_regs2_a, atx_regs2_a) |> 
  mutate(group = fct_inorder(group),
         outcome_label = fct_inorder(outcome_label), 
         interaction_label = fct_inorder(interaction_label)) -> for_figure_1_a

```



@fig-coef-plot shows the estimated effect of an additional piece of good news on average (black dots at top of each panel) and for subgroups defined by pre-treatment covariates for each of the six outcomes in @tbl-outcome-vars. We find no discernible effect of providing more pieces of "good news" on any outcome on average; the lowest p-value is `r ate_regs2 |> pull(p.value) |> min() |> round(3)`. Estimated effects for subgroups defined by age, political ideology, party ID, pre-treatment pessimism, pre-treatment concern about climate change, and pre-treatment self-efficacy are generally small and confidence intervals include zero in all but one case. The general impression is that providing good news about climate mitigation does not affect beliefs, attitudes, or policy preferences.  

```{r fig-coef-plot, echo = F, warning = F, fig.cap = "Effect of 'good news' on six outcomes, on average and across subgroups", fig.height=8, fig.width=8}
#| label: fig-coef-plot

key_estimate <- for_figure_1 |> 
  filter(group == "High" & str_detect(interaction_label, "factual pessimism") & str_detect(outcome_label, "Donation")) |> pull(estimate)

for_figure_1 |> 
  ggplot(aes(x = estimate, xmin = ci_lower, xmax = ci_upper, y = fct_rev(interaction_label), col = group, pch = group)) + # fct_rev(group))) + 
  ggstance::geom_pointrangeh(position = position_dodge(width = -.65)) + 
  geom_vline(xintercept = 0, lty = 2) + 
  labs(y = "", x = "Estimated effect of additional piece of 'good news' on standardized outcome",
       col = "Subgroup", pch = "Subgroup") + 
  scale_color_manual(values = c("black", "#F3AD1E", "#00A08B", "#ED2226")) +
  facet_wrap(vars(outcome_label)) + # , scales = "free_x") + 
  theme_bw() + 
  theme(legend.position = "top") + 
  geom_text(data = filter(for_figure_1 |> mutate(n_label = str_c("n=", n)), group %in% c("All", "Medium")), mapping = aes(x = .22, y = fct_rev(interaction_label), label = n_label), col = "black", size = 2.5)
```

The only exception to the generally null results in @fig-coef-plot is the finding that an additional piece of good news reduces donation amounts for the most pessimistic respondents. The estimated effect for these respondents is `r  key_estimate |> round(3)` standard deviations (about \$`r round(abs(key_estimate) * sd(dat$donate_amount_4, na.rm = T), 2)`), and the interaction term (comparing the effect for this group to the effect for the low-pessimism group) has a p-value of `r round(key_p_value, 3)`. Given that we are testing multiple interaction terms on multiple outcomes, adjustment for multiple testing is appropriate. Specifically, following our PAP we seek a family-wise error rate (FWER) no higher than .05 in assessing the null hypothesis that all interactions[^pap-dev-2] between the number of good news items and terciles of pre-treatment pessimism are zero, separately for the first three outcomes (relating to attitudes and beliefs) and the second three outcomes (relating to policy preferences). We use randomization inference to determine a p-value cutoff that would lead to a false positive rate no higher than .05 across these interactions and outcomes.[^procedure] The resulting p-value cutoff for the policy preference outcomes is `r round(family_B_cutoff_itx, 3)`, which is lower than the raw p-value for this interaction. We therefore fail to reject the null hypothesis that all interactions are zero and the effect of additional good news does not vary across levels of pre-treatment pessimism.[^bayes-addition]

[^pap-dev-2]: Our PAP said we would only include interactions with "top pessimism tercile", but because interactions with "middle pessimism tercile" are relevant we include all interactions. 

[^procedure]: We use the simulation approach outlined in Alexander Coppock, "10 Things To Know About Multiple Comparisons" [(link)](https://egap.org/resource/10-things-to-know-about-multiple-comparisons/). 

[^bayes-addition]: Consistent with this, we find no evidence that good news has a larger effect on pessimistic respondents' optimism about future emissions (top left panel of @fig-coef-plot), and we find no evidence that good news differentially reduces pessimistic respondents' willingness to pay taxes or support aggressive policy. 


<!-- multiple hypothesis test adjustments for info treatments, average effects -->

```{r mht-avg-by-info-treatment, cache = T}
stats_for_dat_info_treatments <- function(dat){
  
  treatments <- c("jobs_info", "us_emissions_info", "solar_cost_info")
  treatment_labels <- c("Renewables\njobs", "US emissions", "Solar cost")
  outcomes <- c("emissions_prediction_standardized", "concern_index_standardized", "efficacy_standardized", "wtp_taxes_standardized", "ira_response_standardized", "donate_amount_standardized")
  outcome_labels = c("DV: Optimism about\nfuture emissions", "DV: Concern about\nclimate change", "DV: Efficacy index", "DV: Willingness to\npay higher taxes", "DV: Support for\naggressive policy", "DV: Donation") # c("Optimism about future emissions", "Concern", "Efficacy", "Willingness to pay higher taxes", "Support for aggressive US policy", "Donation")
  controls <- c("as.integer(concern_cc_pre) + as.integer(lib_con)", "as.integer(concern_cc_pre) + as.integer(lib_con)", "as.integer(efficacy_pre) + as.integer(lib_con)", "as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing", "as.integer(concern_cc_pre) + as.integer(lib_con)","as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing")

  expand_grid(tibble(treatment = treatments,
                   treatment_label = treatment_labels),
            tibble(outcome = outcomes,
                   outcome_label = outcome_labels,
                   controls = controls)) |> 
  mutate(the_formula = map(str_c(outcome, " ~ ", treatment, " + ", controls), as.formula),
         reg = map(the_formula, ~ estimatr::lm_robust(formula = .x, data = dat)),
         tidied = map(reg, broom::tidy)) -> info_treatment_regs2 

 info_treatment_regs2 |> 
   select(-outcome) |> 
   unnest(tidied) |> 
   filter(str_detect(term, "included"))  |> 
   select(outcome_label, treatment_label, estimate, p.value)
}  

# stats_for_dat_info_treatments(dat) |> filter(p.value < .05)

set.seed(12345)
tibble(j = 1:1000) |> 
  # reshuffle treatments individually
  mutate(this_dat = map(j, ~ mutate(dat, 
                                    jobs_info = sample(jobs_info),
                                    solar_cost_info = sample(solar_cost_info),
                                    us_emissions_info = sample(us_emissions_info))),
         sfd = map(this_dat, stats_for_dat_info_treatments)) |>
  select(-this_dat) |> 
  unnest(sfd) |> 
  mutate(family = ifelse(outcome_label %in% c("DV: Optimism about\nfuture emissions", "DV: Concern about\nclimate change", "DV: Efficacy index"), "A", "B")) -> longer

# choose a t-stat or p-value cutoff. count how many family-wise rejections we get 
longer |> 
  expand_grid(p_star = seq(.001, .01, length = 100)) |> 
  arrange(p_star, j) |> 
  group_by(j, family, p_star) |> 
  nest() |> 
  mutate(reject = map2_lgl(data, p_star, reject_or_nah)) |> 
  group_by(p_star, family) |> 
  summarize(reject_rate = mean(reject)) -> summarized_info_treatments 

family_A_cutoff_info <- summarized_info_treatments  |> 
  ungroup() |> 
  filter(family == "A" & reject_rate < .05) |> 
  arrange(desc(p_star)) |> 
  slice(1) |> 
  pull(p_star)

family_B_cutoff_info <- summarized_info_treatments  |> 
  ungroup() |> 
  filter(family == "B" & reject_rate < .05) |> 
  arrange(desc(p_star)) |> 
  slice(1) |> 
  pull(p_star)
```

<!-- multiple hypothesis test adjustments for info treatments, interactions -->
```{r mht-itx-by-info-treatments, cache = T}
stats_for_dat_info_treatments_itx <- function(dat){
  
  expand_grid(tibble(treatment = c("jobs_info", "us_emissions_info", "solar_cost_info"),
                     treatment_label = c("Renewables\njobs", "US emissions", "Solar cost")),
              tibble(interaction_variable = "pessimism_tercile",
                     interaction_label = "By pre-treatment\nfactual pessimism",
                     group_names = list(c("Low", "Medium", "High"))),
              tibble(outcome = c("emissions_prediction_standardized", "concern_index_standardized", "efficacy_standardized", "wtp_taxes_standardized", "ira_response_standardized", "donate_amount_standardized"),
                     outcome_label = c("DV: Optimism about\nfuture emissions", "DV: Concern about\nclimate change", "DV: Efficacy index", "DV: Willingness to\npay higher taxes", "DV: Support for\naggressive policy", "DV: Donation"),
                     controls = c("as.integer(concern_cc_pre) + as.integer(lib_con)", "as.integer(concern_cc_pre) + as.integer(lib_con)", "as.integer(efficacy_pre) + as.integer(lib_con)", "as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing", "as.integer(concern_cc_pre) + as.integer(lib_con)","as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing"))) |> 
    mutate(the_formula = map(str_c(outcome, " ~ ", treatment, "*", interaction_variable, " + ", controls), as.formula),
         reg = map(the_formula, ~ estimatr::lm_robust(formula = .x, data = dat)),
         tidied = map(reg, broom::tidy)) |>
    select(-outcome) |> 
    unnest(tidied) |> 
    filter(str_detect(term, "included:pessimism_tercile")) |> 
    select(outcome_label, treatment_label, term, estimate, p.value)
}
    
set.seed(12345)
tibble(j = 1:1000) |> 
  # reshuffle treatment
  mutate(this_dat = map(j, ~ mutate(dat, 
                                    jobs_info = sample(jobs_info),
                                    solar_cost_info = sample(solar_cost_info),
                                    us_emissions_info = sample(us_emissions_info))),
         sfd = map(this_dat, stats_for_dat_info_treatments_itx)) |>
  select(-this_dat) |> 
  unnest(sfd) |> 
  mutate(family = ifelse(outcome_label %in% c("DV: Optimism about\nfuture emissions", "DV: Concern about\nclimate change", "DV: Efficacy index"), "A", "B")) -> longer

# choose a t-stat or p-value cutoff. count how many family-wise rejections we get, averaging across all tests for the outcome family 
longer |> 
  expand_grid(p_star = seq(.001, .01, length = 100)) |> 
  arrange(p_star, j) |> 
  group_by(j, family, p_star) |> 
  nest() |> 
  mutate(reject = map2_lgl(data, p_star, reject_or_nah)) |> 
  group_by(p_star, family) |> 
  summarize(reject_rate = mean(reject)) -> summarized_info_treatments_itx 

# stats_for_dat_info_treatments_itx(dat) |> filter(p.value < .05)

family_A_cutoff_info_itx <- summarized_info_treatments_itx  |> 
  ungroup() |> 
  filter(family == "A" & reject_rate < .05) |> 
  arrange(desc(p_star)) |> 
  slice(1) |> 
  pull(p_star)

family_B_cutoff_info_itx <- summarized_info_treatments_itx  |> 
  ungroup() |> 
  filter(family == "B" & reject_rate < .05) |> 
  arrange(desc(p_star)) |> 
  slice(1) |> 
  pull(p_star)
```


<!-- now the regressions for the info treatment figures -- ideally combined with the MHT analysis, but not so here -->

```{r pre-reg-regressions-info-treatments, echo = F, message = F}
treatments <- c("jobs_info", "us_emissions_info", "solar_cost_info", "gas_tax_info", "FL_39_DO")
treatment_labels <- c("Renewables\njobs", "US emissions", "Solar cost", "Gas tax\ncomparison", "Inflation\nReduction Act")
outcomes <- c("emissions_prediction_standardized", "concern_index_standardized", "efficacy_standardized", "wtp_taxes_standardized", "ira_response_standardized", "donate_amount_standardized")
outcome_labels = c("DV: Optimism about\nfuture emissions", "DV: Concern about\nclimate change", "DV: Efficacy index", "DV: Willingness to\npay higher taxes", "DV: Support for\naggressive policy", "DV: Donation") # c("Optimism about future emissions", "Concern", "Efficacy", "Willingness to pay higher taxes", "Support for aggressive US policy", "Donation")
controls <- c("as.integer(concern_cc_pre) + as.integer(lib_con)", "as.integer(concern_cc_pre) + as.integer(lib_con)", "as.integer(efficacy_pre) + as.integer(lib_con)", "as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing", "as.integer(concern_cc_pre) + as.integer(lib_con)","as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing")

expand_grid(tibble(treatment = treatments,
                   treatment_label = treatment_labels),
            tibble(outcome = outcomes,
                   outcome_label = outcome_labels,
                   controls = controls)) |> 
  mutate(the_formula = map(str_c(outcome, " ~ ", treatment, " + ", controls), as.formula),
         reg = map(the_formula, ~ estimatr::lm_robust(formula = .x, data = dat))) -> info_treatment_regs2 

info_treatment_regs2 |> 
  mutate(tidied = map(reg, broom::tidy),
         n = map_int(reg, nobs),
         treatment_label = fct_inorder(treatment_label),
         outcome_label = fct_inorder(outcome_label)) |> 
  select(-outcome) |> 
  unnest(tidied) |> 
  filter(str_detect(term, treatment)) |> 
  filter(!(treatment_label == "Inflation\nReduction Act" & outcome_label %in% c("DV: Support for\naggressive policy", "DV: Donation"))) -> for_info_treatment_plot

sfd_info_treatments <- stats_for_dat_info_treatments(dat)
```


```{r}
expand_grid(tibble(treatment = treatments,
                   treatment_label = treatment_labels),
            tibble(interaction_variable = interaction_variables[4],
                   interaction_label = interaction_labels[4],
                   group_names = list(c("Low", "Medium", "High"))),
            tibble(outcome = outcomes,
                   outcome_label = outcome_labels,
                   controls = controls)) |> 
  mutate(the_formula = map(str_c(outcome, " ~ ", treatment, "*", interaction_variable, " + ", controls), as.formula),
         reg = map(the_formula, ~ estimatr::lm_robust(formula = .x, data = dat))) -> info_treatment_itx_regs 
  
info_treatment_itx_regs |> 
  mutate(tg = map2(reg, group_names, tercile_gaps),
         n = map_int(reg, nobs)) |> 
  unnest(tg) |> 
  select(treatment_label, estimate, ci_lower, ci_upper, interaction_label, group, outcome_label, n) |> 
  filter(!(treatment_label == "Inflation\nReduction Act" & outcome_label %in% c("DV: Support for\naggressive policy", "DV: Donation"))) |>
  mutate(treatment_label = fct_inorder(treatment_label),
         outcome_label = fct_inorder(outcome_label),
         group = recode(group, !!!list("High" = "High pessimism", "Medium" = "Medium pessimism", "Low" = "Low pessimism")),
         group = factor(group, levels = c("High pessimism", "Medium pessimism", "Low pessimism"))) -> for_info_treatment_itx_plot 

sfd_info_treatments_itx <- stats_for_dat_info_treatments_itx(dat)
```



@fig-info-treatments reports the effect of each information treatment separately on each outcome, both on average and by respondent factual pessimism.[^appx-extras] Considering each test individually, we find that providing the "Renewables jobs" treatment reduces willingness to pay higher taxes by about `r  sfd_info_treatments |> filter(p.value < .05) |> pull(estimate) |> abs() |> round(2)` standard deviations across the whole sample (p-value: `r sfd_info_treatments |> filter(p.value < .05) |> pull(p.value) |> round(3)`); all other estimates of average effects are not significant at the .05 level. Following the simulation procedure stated in our PAP, we determine that to obtain a FWER of .05 for all three information treatments across the policy preference outcomes (willingness to pay taxes, support for aggressive policy, and donation), we should apply a p-value threshold of `r round(family_B_cutoff_info, 3)`. We therefore fail to reject the null hypothesis that there is no effect of any information treatment on any of our policy preference outcomes. Turning to treatment effect heterogeneity, and again judging by raw p-values, we find two significant interactions: the "Renewables jobs" treatment increases support for aggressive policy more among the high pessimism group compared to the low pessimism group (p-value: `r sfd_info_treatments_itx |> filter(str_detect(outcome_label, "aggressive") & p.value < .05) |> pull(p.value) |> round(3)`) and the "Solar cost" treatment increases willingness to pay higher taxes more among the middle pessimism group compared to the low pessimism group (p-value: `r sfd_info_treatments_itx |> filter(str_detect(outcome_label, "higher taxes") & p.value < .05) |> pull(p.value) |> round(3)`). But our simulation procedure indicates a p-value threshold of `r round(family_B_cutoff_info_itx, 3)` to maintain a FWER of .05 across interactions and treatments for the three policy preference outcomes, so again we fail to reject the null hypothesis that there is no differential effect of any information treatment on any of our policy preference outcomes.   

[^appx-extras]: Survey respondents were exposed to two information treatments not highlighted in @fig-coef-plot but discussed in our PAP: half of respondents were given the answer to the gas tax comparison question on the knowledge quiz, and half were told about the IRA before the other outcome variables. To simplify presentation (allowing us to focus on the effect of providing "good news") <!-- new -->  we omit these treatments from the paper, but we discuss and present the results (all null after corrections for multiple testing) in the Appendix (@fig-info-treatments-all).




```{r, fig.height=6, fig.width=6.5}
#| label: fig-info-treatments
#| fig-cap: Effect of each information treatment, on average and by respondent pre-treatment factual pessimism

for_info_treatment_plot |> 
  filter(!treatment_label %in% c("Inflation\nReduction Act", "Gas tax\ncomparison")) |> 
  select(-treatment, -the_formula, -reg, -controls, -term) |> 
  mutate(group = "All") |> 
  bind_rows(for_info_treatment_itx_plot |> select(-interaction_label) |> rename(conf.high = ci_upper, conf.low = ci_lower)) |> 
  mutate(group = fct_inorder(group),
         this_alpha = ifelse(group %in% c("All", "Medium pessimism"), 1, 0)) -> for_info_treatment_plot2

for_info_treatment_plot2 |> 
#   mutate(outcome_label = str_c(outcome_label, " (n=", n, ")")) |> 
  filter(!treatment_label %in% c("Gas tax\ncomparison", "Inflation\nReduction Act")) |> 
  ggplot(aes(x = estimate, xmin = conf.low, xmax = conf.high, y = fct_rev(treatment_label), col = group, pch = group)) + 
  ggstance::geom_pointrangeh(position = position_dodge(width = -.5)) + 
  geom_vline(xintercept = 0, lty = 2) + 
  labs(y = "Information treatment", x = "Estimated effect of providing treatment on standardized outcome",
       col = "Subgroup", pch = "Subgroup") + 
  scale_color_manual(values = c("black", "#F3AD1E", "#00A08B", "#ED2226")) +
  # scale_color_manual(values = c("black", "#ED2226", "#D522ED",  "#222CED")) +
  facet_wrap(vars(outcome_label)) + 
  theme_bw() + 
  theme(legend.position = "top") + 
  expand_limits(x = c(-.32, .32)) + 
  geom_text(data = for_info_treatment_plot2 |> filter(!treatment_label %in% c("Gas tax\ncomparison", "Inflation\nReduction Act")) |> mutate(n_label = str_c("n=", n)), mapping = aes(x = -.275, y = fct_rev(treatment_label), label = n_label, alpha = this_alpha), col = "black", size = 2.5, position = position_dodge(width = -.5), show.legend = F)


```

A power analysis explained and presented in the Appendix indicates that our design is likely to detect substantively meaningful effects of good news. If each piece of good news increased all six outcomes by as little as .05 standard deviations, then we would reject the null hypothesis of no effect with probability between .5 and .95 depending on the outcome; taking into account multiple comparisons, we would reject the null hypothesis of no effect on any outcome with probability .95 for the first three outcomes and probability .8 for the second three outcomes. Likewise, if the effect of each piece of good news differed by .1 standard deviations between high- and low-pessimism respondents, we would reject the null of no treatment effect heterogeneity with probability around .5 across outcomes, and with substantially higher probability when we consider families of outcomes and take into account multiple comparisons. 

An important explanation for our null findings is that our information treatments appear not to have affected respondents' confidence in the success of future mitigation efforts, as shown by the null effects of these treatments on the "Optimism about future emissions" measure both on average and in subgroups. Our information treatments were designed to increase respondents' optimism about the possibility of limiting the effects of climate change, but they apparently did not do so.[^us-focused] In the absence of such an effect, it is not surprising that we also find no effects of these information treatments on other attitudes and preferences. In this sense, one key conclusion of our study is that Americans' perceptions of the tractability of climate mitigation are resistant to change by factual information treatments. 

[^us-focused]:  It is possible that the US-focused facts we provided made respondents more optimistic about US mitigation efforts without changing their view of the prospects for mitigation globally. Future research could examine whether news of domestic vs. global progress has different effects. <!-- new -->  

```{r}
#| eval: false
dat |> 
  group_by(n_good_news_wo_ira) |>
  count(ep = as.integer(emissions_prediction)) |> 
  mutate(prop = n/sum(n)) |> 
  ggplot(aes(x = ep, y = prop, col = as.factor(n_good_news_wo_ira))) + 
  geom_point() + 
  geom_line(aes(group = n_good_news_wo_ira)) + 
  
```



## Discussion

Using an "ask-tell" design to provide American survey respondents factual updates on climate change mitigation, we find no effect of "good news" on either attitudes or policy preferences among US respondents. <!-- new --> These findings contrast with those of studies that show that emphasizing progress in addressing climate change can demotivate survey respondents [@hornsey2016cautionary] as well as studies that show that optimistic reframings of climate change can reassure and motivate survey respondents [e.g. @bain2012promoting; @dasandi2022positive]. Our findings are more consistent with @bernauer2016simple and @ettinger2021climate, who find limited impacts of such reframings.  

Although our null results do not support the idea that broader recognition of progress in climate mitigation will inspire greater action, they also do not support the idea that broader recognition will reduce support for decarbonization. In that sense, our results suggest that public support for mitigation is more durable than previous studies suggest.

Our results are of course specific to the US in 2023, a setting in which opinion on climate change and climate policy is seen as stable and highly politicized [@egan2017climate]. More research is necessary to determine how factual messages emphasizing climate progress affect public opinion in other settings. 

Moreover, while this study provides evidence that factual information about progress in climate mitigation has limited effects on respondent attitudes, we consider as unresolved the question of how the perceived tractability of climate change mitigation affects individuals' willingness to take action. We had hoped to experimentally vary these perceptions by exposing participants to surprising but factual "good news" about recent progress in decarbonization, but perceptions of tractability were resistant to change. Future studies that investigate these questions should therefore focus on locating information treatments that have a stronger impact on these perceptions of tractability or subgroups whose perceptions are more easily manipulated.


{{< pagebreak >}}

## References

::: {#refs}
:::

{{< pagebreak >}}

## Appendix

### Deviations from the pre-analysis plan (PAP)

The analysis in the paper differs from the PAP in the following ways (all mentioned in the main body of the paper): 

- Unlike in the PAP, we standardize outcome variables. This allows for simpler presentation.
- In seeking to adjust our hypothesis tests of treatment effect heterogeneity in the paper, we focus on the null hypothesis that all interactions between the moderator (pre-treatment pessimism) and the treatment (number of good news items) are zero. In the PAP, the null hypothesis only relates to interactions involving the indicator for "highest tercile of pre-treatment pessimism". Given that interactions involving the middle category are also of interest (and our theoretical expectations allow for non-monotonic relationships between pre-treatment pessimism and the effect of good news on beliefs), this seemed more appropriate.
- In our PAP, we said that analysis of individual information treatments would include two treatments that are not well described as "good news" -- one information treatment about the US gas tax relative to OECD counterparts, and one question order treatment (whether the IRA is described before or after other outcome questions). The null results produced by this analysis are fully reported in the appendix, though these treatments are not included in the reported adjustments for multiple comparisons). We thought it was better to not include these treatments in the main body of the paper to keep the description of the design simple.

{{< pagebreak>}} 

### Attention check and analysis omitting respondents who fail

The attention check appeared in the middle of the section in which we measured outcomes. (If an attention check is used to exclude inattentive respondents, it is better for it to appear before treatments are applied. This placement is better for assessing how attentive respondents are when outcomes are measured.) The question states, "Some people argue that the risks from climate change have been widely exaggerated. Also, some researchers believe that the responses that come from surveys like this one are not reliable. To show skeptics that respondents to this survey do pay attention, could you please answer both 'Strongly disagree' and 'Neither agree nor disagree' below?" Response options provided were "Strongly disagree", "Somewhat disagree", "Neither agree nor disagree", "Somewhat agree", "Strongly agree".


@fig-coef-plot-attn reproduces @fig-coef-plot without the respondents who failed the attention check. @tbl-avg-effects-wo-attn-check presents the underlying analysis in table form, to complement @tbl-avg-effects.

```{r for-fig-wo-attn-check-failers}
# the first figure without people who failed attention check
expand_grid(tibble(interaction_variable = interaction_variables,
                   interaction_label = interaction_labels,
                   group_names = list(c("Low", "Medium", "High"))),
            tibble(outcome = outcomes,
                   outcome_label = outcome_labels,
                   controls = controls)) |> 
  mutate(the_formula = map(str_c(outcome, " ~ n_good_news_wo_ira *", interaction_variable, " + ", controls), as.formula),
         reg = map(the_formula, ~ estimatr::lm_robust(formula = .x, data = filter(dat, attention_check == "Strongly disagree,Neither agree nor disagree")))) -> ate_intx_regs_X 

tibble(outcome = outcomes,
       outcome_label = outcome_labels,
       controls = controls) |> 
  mutate(the_formula = map(str_c(outcome, " ~ n_good_news_wo_ira + ", controls), as.formula),
         reg = map(the_formula, ~ estimatr::lm_robust(formula = .x, data = filter(dat, attention_check == "Strongly disagree,Neither agree nor disagree")))) -> ate_regs_X 

ate_intx_regs_X |> 
  mutate(tg = map2(reg, group_names, tercile_gaps),
         n = map_int(reg, nobs)) |> 
  unnest(tg) |> 
  select(estimate, ci_lower, ci_upper, interaction_label, group, outcome_label, n) -> atx_regs2_X 

ate_regs_X |> 
  mutate(tidied = map(reg, broom::tidy),
         n = map_int(reg, nobs)) |> 
  select(-outcome) |> 
  unnest(tidied) |> 
  mutate(group = "All",
         interaction_label = "All") |> 
  filter(term == "n_good_news_wo_ira") |> 
  select(estimate, ci_lower = conf.low, ci_upper = conf.high, interaction_label, group, outcome_label, n) -> ate_regs2_X
  
bind_rows(ate_regs2_X, atx_regs2_X) |> 
  mutate(group = fct_inorder(group),
         outcome_label = fct_inorder(outcome_label),
         interaction_label = fct_inorder(interaction_label)) -> for_plot_X
```



```{r fig-coef-plot-attn, fig.height=7, fig.width=8}
#| label: fig-coef-plot-attn
#| fig-cap: Effect of 'good news' on six outcomes, on average and across subgroups (excluding respondents who fail the attention check)

for_plot_X |> 
  ggplot(aes(x = estimate, xmin = ci_lower, xmax = ci_upper, y = fct_rev(interaction_label), col = group, pch = group)) + # fct_rev(group))) + 
  ggstance::geom_pointrangeh(position = position_dodge(width = -.35)) + 
  geom_vline(xintercept = 0, lty = 2) + 
  labs(y = "", x = "Estimated effect of additional piece of 'good news' on standardized outcome",
       col = "Subgroup", pch = "Subgroup") + 
  scale_color_manual(values = c("black", "#F3AD1E", "#00A08B", "#ED2226")) +
  facet_wrap(vars(outcome_label)) + # , scales = "free_x") + 
  theme_bw() + 
  theme(legend.position = "top") + 
  geom_text(data = filter(for_plot_X |> mutate(n_label = str_c("n=", n)), group %in% c("All", "Medium")), mapping = aes(x = .25, y = fct_rev(interaction_label), label = n_label), col = "black", size = 2.5)
```


```{r pre-reg-regressions-ate-2, echo = F, message = F}
dat2 <- dat |> filter(attention_check == "Strongly disagree,Neither agree nor disagree")

# hypothesis family A regressions
efficacy_reg2 <- estimatr::lm_robust(efficacy_standardized ~ n_good_news_wo_ira + as.integer(efficacy_pre) + as.integer(lib_con), data = dat2)
emissions_reg2 <- estimatr::lm_robust(emissions_prediction_standardized ~ n_good_news_wo_ira + as.integer(concern_cc_pre) + as.integer(lib_con), data = dat2)
concern_reg2 <- estimatr::lm_robust(concern_index_standardized ~ n_good_news_wo_ira + as.integer(concern_cc_pre) + as.integer(lib_con), data = dat2)
# hypothesis family B regressions
wtp_reg2 <- estimatr::lm_robust(wtp_taxes_standardized ~ n_good_news_wo_ira + as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing, data = dat2)
ira_reg2 <- estimatr::lm_robust(ira_response_standardized ~ n_good_news_wo_ira + as.integer(concern_cc_pre) + as.integer(lib_con), data = dat2)
donate_reg2 <- estimatr::lm_robust(donate_amount_standardized ~ n_good_news_wo_ira + as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing, data = dat2)
```

```{r ate-reg-table-2, echo = F}
#| label: tbl-avg-effects-wo-attn-check
#| tbl-cap: Average effect of additional pieces of good news (excluding respondents who failed the attention check)

combined_regs2 <- modelsummary::modelsummary(models = list("Optimism about future emissions" = emissions_reg2,
                                                              "Concern" = concern_reg2,
                                                              "Efficacy" = efficacy_reg2,
                                                              "Willingness to pay higher taxes" = wtp_reg2,
                                                              "Support for more aggressive policy" = ira_reg2,
                                                              "Donation" = donate_reg2),
                           coef_map = c("(Intercept)" = "Intercept",
                                        "n_good_news_wo_ira" = "Good news items provided",
                                        "as.integer(concern_cc_pre)" = "Pre-treatment climate concern",
                                        "as.integer(efficacy_pre)" = "Pre-treatment self-efficacy",
                                        "as.integer(lib_con)" = "Pre-treatment liberal-ness",
                                        "as.integer(hh_income)" = "Household income category",
                                        "hh_income_missing" = "Household income missing"
                                        ),
                           #title = "Average effect of additional pieces of good news (excluding respondents who failed the attention check)",
                           gof_map = c("nobs", "r.squared"),
                           stars = T) |> 
  kableExtra::add_header_above(c(" ", "A: Attitudes/beliefs" = 3, "B: Policy preferences" = 3))

combined_regs2 |> 
  kableExtra::column_spec(1, width = "15em") |> 
  kableExtra::column_spec(2:7, width = "4em") |> 
  kableExtra::kable_styling(latex_options = c("hold_position", "scale_down"), font_size = 9)
```


{{< pagebreak>}} 

### Full set of knowledge quiz questions

```{r tbl-quiz-questions-appx, echo=FALSE}
#| label: tbl-quiz-questions-appx
#| tbl-cap: All questions on our knowledge survey

knowledge_quiz |> 
#   slice(8:10) |> 
  kableExtra::kbl(booktabs = T) |> 
  kableExtra::column_spec(1, width = "25em") |>   # 17
  kableExtra::column_spec(2, width = "10em") |>  # 10
  kableExtra::column_spec(3, width = "16em") |>    # 17
  kableExtra::kable_styling(font_size = 9, latex_options = c("striped", "scale_down")) 

```


{{< pagebreak >}}


<!-- Power analysis for main effects --> 
### Power analysis

@fig-power-individual and @fig-power-fwer show power curves for individual outcomes and collections of outcomes (respectively) for the analysis of average effects in @fig-coef-plot. To produce the figure, we repeat the following procedure 500 times for each value of $\beta \in \{0, .02, .04, .06, .08, .1\}$: 

- reshuffle the assignment of number of pieces of good news, $D_i \in \{0, 1, 2, 3\}$
- replace each observed outcome $Y_i^{\text{obs}}$ with $Y_i = Y_i^{\text{obs}} + \beta D_i$
- run the pre-specified analysis measuring the effect of $D_i$ on $Y_i$

In @fig-power-individual we report, for each of the outcome variables, the proportion of the 500 iterations in which we reject the null that the slope coefficient on $D_i$ is zero (using a p-value threshold of .05). Power varies across outcome variables, but if the effect of a single piece of good news on each outcome ($\beta$) is .075 standard deviations of the outcome (i.e. providing all three pieces vs. 0 would change the outcome by .225 standard deviations), then power for all six outcomes would be above .8.


```{r power-analysis, eval = T, cache = T}
adjust_dat_w_beta <- function(dat, betah){
  dat |> 
    mutate(emissions_prediction_standardized = emissions_prediction_standardized + betah*n_good_news_wo_ira,
           efficacy_standardized = efficacy_standardized + betah*n_good_news_wo_ira,
           concern_index_standardized = concern_index_standardized + betah*n_good_news_wo_ira, 
           wtp_taxes_standardized = wtp_taxes_standardized + betah*n_good_news_wo_ira, 
           ira_response_standardized = ira_response_standardized + betah*n_good_news_wo_ira,
           donate_amount_standardized = donate_amount_standardized + betah*n_good_news_wo_ira)
}

set.seed(12345)
expand_grid(j = 1:500,
            betah = c(0, .02, .04, .06, .08, .1)) |> 
  # reshuffle treatment but also add a treatment effect
  mutate(this_dat = map(betah, ~ mutate(dat, n_good_news_wo_ira = sample(n_good_news_wo_ira))),
         this_dat = map2(this_dat, betah, adjust_dat_w_beta),
         sfd = map(this_dat, stats_for_dat)) |>
  select(-this_dat) |> 
  unnest(sfd) |> 
  mutate(family = ifelse(outcome_label %in% c("DV: Optimism about\nfuture emissions", "DV: Concern about\nclimate change", "DV: Efficacy index"), "A: beliefs and attitudes", "B: policy preferences")) |> 
  mutate(outcome_label = fct_inorder(outcome_label)) -> powa_longer
```


```{r fig-power-individual, fig.cap = "Power curve for coefficient on good news items, by outcome variable"}
#| label: fig-power-individual

# power for each outcome individually
powa_longer |> 
  group_by(betah, outcome_label) |> 
  summarize(power = mean(p.value < .05)) |> 
  mutate(outcome_label_x = fct_inorder(str_replace(as.character(outcome_label), "DV: ", ""))) |> 
  ggplot(aes(x = betah, y = power, col = outcome_label_x)) + 
  geom_line() + 
  labs(x = "Effect of single good news item on outcome\n(in std. devs. of outcome variable)",
       y = "Estimated power",
       col = "Outcome variable") + 
  geom_hline(yintercept = c(.05, .8), lty = c(2, 3)) +
  theme_bw()
```

In @fig-power-fwer we report, for each family of outcome variables, the proportion of the 500 iterations in which we reject the null that all of the slope coefficients in the family are zero (using the p-value threshold we determined would achieve a FWER no greater than .05, which is `r round(family_A_cutoff, 4)` for hypothesis family A and `r round(family_B_cutoff, 4)` for hypothesis family B). If the effect of a single piece of good news on each outcome ($\beta$) were .05 standard deviations of the outcome (i.e. providing all three pieces vs. 0 would change the outcome by .15 standard deviations), then power for both families would be .8. Note that power when $\beta = 0$ is around .05, as intended.


```{r fig-power-fwer, fig.cap = "Power curve for coefficient on good news items, by family of outcome variables"}
#| label: fig-power-fwer

powa_longer |> 
  mutate(p_value_cutoff = ifelse(family == "A: beliefs and attitudes", family_A_cutoff, family_B_cutoff)) |> 
  group_by(betah, family, j) |> 
  summarize(rejected = sum(p.value < p_value_cutoff) > 0) |>
  group_by(betah, family) |> 
  summarize(power = mean(rejected)) |> 
  ggplot(aes(x = betah, y = power, col = family)) + 
  geom_line() + 
  labs(x = "Effect of single good news item on outcome\n(in std. devs. of outcome variable)",
       y = "Estimated power",
       col = "Family of outcome\nvariables") + 
  geom_hline(yintercept = c(.05, .8), lty = c(2, 3)) + 
  theme_bw()

```


@fig-power-individual-itx and @fig-power-fwer-itx show power curves for individual outcomes and collections of outcomes (respectively) for the analysis of heterogeneity in the effect of additional pieces of "good news" by pre-treatment factual pessimism (reported in @fig-coef-plot). To produce the figure, we repeat the following procedure 500 times for each value of $x \in \{0, .02, .04, .06, .08, .1\}$: 

- reshuffle the assignment of number of pieces of good news, $D_i \in \{0, 1, 2, 3\}$
- replace each observed outcome $Y_i^{\text{obs}}$ with $Y_i = Y_i^{\text{obs}} + \beta_i D_i$, where $\beta_i$ is $-x/2$ for respondents in the lowest pessimism tercile, 0 for respondents in the middle pessimism tercile, and $x/2$ for respondents in the highest pessimism tercile (so that $x$ is the difference in the effect of $D_i$ between respondents in the highest and lowest tercile)
- run the pre-specified analysis measuring the effect of $D_i$ interacted with pessimism tercile on $Y_i$


```{r power-analysis-itx, cache = T}
adjust_dat_w_beta_itx <- function(dat, x){
  dat |> 
    mutate(beta_i = (as.integer(pessimism_tercile) - 2)*x,
           emissions_prediction_standardized = emissions_prediction_standardized + beta_i*n_good_news_wo_ira,
           efficacy_standardized = efficacy_standardized + beta_i*n_good_news_wo_ira,
           concern_index_standardized = concern_index_standardized + beta_i*n_good_news_wo_ira, 
           wtp_taxes_standardized = wtp_taxes_standardized + beta_i*n_good_news_wo_ira, 
           ira_response_standardized = ira_response_standardized + beta_i*n_good_news_wo_ira,
           donate_amount_standardized = donate_amount_standardized + beta_i*n_good_news_wo_ira)
}

set.seed(12345)
expand_grid(j = 1:500,
            x = c(0, .02, .04, .06, .08, .1)) |> 
  # reshuffle treatment but also add a treatment effect
  mutate(this_dat = map(x, ~ mutate(dat, n_good_news_wo_ira = sample(n_good_news_wo_ira))),
         this_dat = map2(this_dat, x, adjust_dat_w_beta_itx),
         sfd = map(this_dat, stats_for_dat_intx)) |>
  select(-this_dat) |> 
  unnest(sfd) |> 
  mutate(family = ifelse(outcome_label %in% c("DV: Optimism about\nfuture emissions", "DV: Concern about\nclimate change", "DV: Efficacy index"), "A: beliefs and attitudes", "B: policy preferences")) |> 
  mutate(outcome_label = fct_inorder(outcome_label)) -> powa_longer_itx
```


In @fig-power-individual-itx we report, for each of the outcome variables, the proportion of the 500 iterations in which we reject the null that the slope coefficient on $D_i$ is zero (using a p-value threshold of .05). Across all outcomes, if the effect of a single piece of good news varies by .1 standard deviation between high- and low-pessimism respondents, we obtain a power of around .5. 


```{r fig-power-individual-itx, fig.cap = "Power curve for interactions with pessimism tercile, by outcome variable"}
#| label: fig-power-individual-itx

# power for each outcome individually
powa_longer_itx |> 
  group_by(x, outcome_label) |> 
  summarize(power = mean(p.value < .05)) |> 
  mutate(outcome_label_x = fct_inorder(str_replace(as.character(outcome_label), "DV: ", ""))) |> 
  ggplot(aes(x = x, y = power, col = outcome_label_x)) + 
  geom_line() + 
  labs(x = "Diff. in effect of single good news item on outcome\nbtw high and low pessimism respondents\n(in std. devs. of outcome variable)",
       y = "Estimated power",
       col = "Outcome variable") + 
  geom_hline(yintercept = c(.05, .8), lty = c(2, 3)) +
  theme_bw()
```
In @fig-power-fwer-itx we report, for each family of outcome variables, the proportion of the 500 iterations in which we reject the null that all of the interaction coefficients in the family are zero (using the p-value threshold we determined would achieve a FWER no greater than .05, which is `r round(family_A_cutoff_itx, 4)` for hypothesis family A and `r round(family_B_cutoff_itx, 4)` for hypothesis family B). Here the horizontal axis denotes $x$, the difference in the effect between high- and low-pessimism terciles. Note again that power when $x = 0$ is around .05, as intended.



```{r fig-power-fwer-itx}
#| label: fig-power-fwer-itx
#| fig-cap: Power curve for interactions with pessimism tercile, by family of outcome variables

powa_longer_itx |> 
  mutate(p_value_cutoff = ifelse(family == "A: beliefs and attitudes", family_A_cutoff_itx, family_B_cutoff_itx)) |> 
  group_by(x, family, j) |> 
  summarize(rejected = sum(p.value < p_value_cutoff) > 0) |>
  group_by(x, family) |> 
  summarize(power = mean(rejected)) |> 
  ggplot(aes(x = x, y = power, col = family)) + 
  geom_line() + 
  labs(x = "Diff. in effect of single good news item on outcome\nbtw high and low pessimism respondents\n(in std. devs. of outcome variable)",
       y = "Estimated power",
       col = "Family of outcome\nvariables") + 
  geom_hline(yintercept = c(.05, .8), lty = c(2, 3)) + 
  theme_bw()
```


{{< pagebreak >}}


### Regression tables supporting figures in paper

- @tbl-avg-effects supports estimates of average effects in @fig-coef-plot
- @tbl-by-age supports estimates of heterogeneity by age group in @fig-coef-plot
- @tbl-by-ideology supports estimates of heterogeneity by political ideology group in @fig-coef-plot
- @tbl-by-partyid supports estimates of heterogeneity by party ID group in @fig-coef-plot
- @tbl-by-pessimism supports estimates of heterogeneity by pre-treatment factual pessimism group in @fig-coef-plot
- @tbl-by-concern supports estimates of heterogeneity by pre-treatment concern group in @fig-coef-plot
- @tbl-by-efficacy supports estimates of heterogeneity by pre-treatment efficacy group in @fig-coef-plot

```{r pre-reg-regressions-ate, echo = F, message = F}
# hypothesis family A regressions
efficacy_reg <- estimatr::lm_robust(efficacy_standardized ~ n_good_news_wo_ira + as.integer(efficacy_pre) + as.integer(lib_con), data = dat)
emissions_reg <- estimatr::lm_robust(emissions_prediction_standardized ~ n_good_news_wo_ira + as.integer(concern_cc_pre) + as.integer(lib_con), data = dat)
concern_reg <- estimatr::lm_robust(concern_index_standardized ~ n_good_news_wo_ira + as.integer(concern_cc_pre) + as.integer(lib_con), data = dat)
# hypothesis family B regressions
wtp_reg <- estimatr::lm_robust(wtp_taxes_standardized ~ n_good_news_wo_ira + as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing, data = dat)
ira_reg <- estimatr::lm_robust(ira_response_standardized ~ n_good_news_wo_ira + as.integer(concern_cc_pre) + as.integer(lib_con), data = dat)
donate_reg <- estimatr::lm_robust(donate_amount_standardized ~ n_good_news_wo_ira + as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing, data = dat)
```


```{r tbl-avg-effects, echo = F}
#| tbl-cap: Average effect of additional pieces of good news
#| label: tbl-avg-effects
combined_regs <- modelsummary::modelsummary(models = list("Optimism about future emissions" = emissions_reg,
                                                              "Concern" = concern_reg,
                                                              "Efficacy" = efficacy_reg,
                                                              "Willingness to pay higher taxes" = wtp_reg,
                                                              "Support for more aggressive policy" = ira_reg,
                                                              "Donation" = donate_reg),
                           coef_map = c("(Intercept)" = "Intercept",
                                        "n_good_news_wo_ira" = "Good news items provided",
                                        "as.integer(concern_cc_pre)" = "Pre-treatment climate concern",
                                        "as.integer(efficacy_pre)" = "Pre-treatment self-efficacy",
                                        "as.integer(lib_con)" = "Pre-treatment liberal-ness",
                                        "as.integer(hh_income)" = "Household income category",
                                        "hh_income_missing" = "Household income missing"
                                        ),
                          #  title = "Average effect of additional pieces of good news",
                           gof_map = c("nobs", "r.squared"),
                           stars = T) |> 
  kableExtra::add_header_above(c(" ", "A: Attitudes/beliefs" = 3, "B: Policy preferences" = 3))

combined_regs |> 
  kableExtra::column_spec(1, width = "15em") |> 
  kableExtra::column_spec(2:7, width = "4em") |> 
  kableExtra::kable_styling(latex_options = c("hold_position", "scale_down"), font_size = 9)

```


```{r het-reg-fn}
# function for making table showing the interaction regressions
het_reg_table <- function(intx_var = "pessimism_tercile", coef_map_part = c(
  "intx_var.L" = "Medium pessimism",
  "intx_var.Q" = "High pessimism",
  "n_good_news_wo_ira:intx_var.L" = "Good news X medium pessimism",
  "n_good_news_wo_ira:intx_var.Q" = "Good news X high pessimism"),
  table_title = "Interactions between good news provision and pre-treatment pessimism of beliefs"){
  
  # create intx variable
  dat_tmp <- dat |> mutate(intx_var = get(intx_var))
  
  # run the regressions
  concern_reg_het <- estimatr::lm_robust(concern_index_standardized ~ n_good_news_wo_ira*intx_var + as.integer(concern_cc_pre) + as.integer(lib_con), data = dat_tmp)
  emissions_reg_het <- estimatr::lm_robust(emissions_prediction_standardized ~ n_good_news_wo_ira*intx_var + as.integer(concern_cc_pre) + as.integer(lib_con), data = dat_tmp)
  efficacy_reg_het <- estimatr::lm_robust(efficacy_standardized ~ n_good_news_wo_ira*intx_var + as.integer(efficacy_pre) + as.integer(lib_con), data = dat_tmp)
  wtp_reg_het <- estimatr::lm_robust(wtp_taxes_standardized ~ n_good_news_wo_ira*intx_var + as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing, data = dat_tmp)
  ira_reg_het <- estimatr::lm_robust(ira_response_standardized ~ n_good_news_wo_ira*intx_var + as.integer(concern_cc_pre) + as.integer(lib_con), data = dat_tmp)
  donate_reg_het <- estimatr::lm_robust(donate_amount_standardized ~ n_good_news_wo_ira*intx_var + as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing, data = dat_tmp)
  
  # extra row 
  rows <- tribble(~term,          ~mod1,  ~mod2, ~mod3, ~mod4, ~mod5, ~mod6,
                'Covariates included', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y') 
  attr(rows, 'position') <- c(13)
  
  # output the table 
  modelsummary::modelsummary(models = list("Optimism about future emissions" = emissions_reg_het,
                                         "Concern" = concern_reg_het,
                                         "Efficacy" = efficacy_reg_het,
                                         "Willingness to pay higher taxes" = wtp_reg_het,
                                         "Support for aggressive US policy" = ira_reg_het,
                                         "Donation" = donate_reg_het),
                           coef_map = c("(Intercept)" = "Intercept",
                                        "n_good_news_wo_ira" = "Good news items provided", coef_map_part),
                           # title = table_title,
                           gof_map = c("nobs", "r.squared"),
                           stars = T,
                           add_rows = rows)  |> 
  kableExtra::add_header_above(c(" ", "A: Attitudes/beliefs" = 3, "B: Policy preferences" = 3)) |> 
  kableExtra::column_spec(1, width = "15em") |> 
  kableExtra::column_spec(2:7, width = "4em") |> 
  kableExtra::kable_styling(latex_options = c("hold_position", "scale_down"), font_size = 9)
}
```


{{< pagebreak >}}


```{r age-intx-table}
#| tbl-cap: Interactions between good news provision and age group
#| label: tbl-by-age

het_reg_table(intx_var = "age_group", 
              coef_map_part = c(
  "intx_var35-49" = "Age 35-49",
  "intx_var50+" = "Age 50+",
  "n_good_news_wo_ira:intx_var35-49" = "Good news X Age 35-49",
  "n_good_news_wo_ira:intx_var50+" = "Good news X Age 50+"))
```

```{r ideology-intx-table}
#| tbl-cap: Interactions between good news provision and ideology
#| label: tbl-by-ideology
het_reg_table(intx_var = "ideology", 
              coef_map_part = c(
  "intx_varModerate" = "Moderate",
  "intx_varLiberal" = "Liberal",
  "n_good_news_wo_ira:intx_varModerate" = "Good news X Moderate",
  "n_good_news_wo_ira:intx_varLiberal" = "Good news X Liberal"))
```


```{r partyid-intx-table}
#| tbl-cap: Interactions between good news provision and party ID
#| label: tbl-by-partyid
het_reg_table(intx_var = "party_id", 
              coef_map_part = c(
  "intx_varIndependent" = "Independent",
  "intx_varDemocrat" = "Democrat",
  "n_good_news_wo_ira:intx_varIndependent" = "Good news X Independent",
  "n_good_news_wo_ira:intx_varDemocrat" = "Good news X Democrat"))
```


```{r pessimism-intx-table}
#| tbl-cap: Interactions between good news provision and pre-treatment pessimism
#| label: tbl-by-pessimism
het_reg_table()
```


```{r concern-intx-table}
#| tbl-cap: Interactions between good news provision and pre-treatment concern
#| label: tbl-by-concern
het_reg_table(intx_var = "concern_group", 
              coef_map_part = c(
  "intx_varModerately concerned" = "Moderately concerned",
  "intx_varVery or extremely concerned" = "Very or extremely concerned",
  "n_good_news_wo_ira:intx_varModerately concerned" = "Good news X Moderately concerned",
  "n_good_news_wo_ira:intx_varVery or extremely concerned" = "Good news X Very or extremely concerned"))
```

```{r efficacy-intx-table}
#| tbl-cap: Interactions between good news provision and pre-treatment efficacy
#| label: tbl-by-efficacy
het_reg_table(intx_var = "efficacy_group", 
              coef_map_part = c(
  "intx_varMedium efficacy" = "Medium efficacy",
  "intx_varHigh efficacy" = "High efficacy",
  "n_good_news_wo_ira:intx_varMedium efficacy" = "Good news X Medium efficacy",
  "n_good_news_wo_ira:intx_varHigh efficacy" = "Good news X High efficacy"))
```

### Results for all information treatments 

@fig-info-treatments-all is a version of @fig-info-treatments presenting results for all information treatments discussed in our PAP including the gas tax treatment (informing respondents that the US's gas tax is the lowest in the OECD) and the IRA treatment (informing respondents about the Inflation Reduction Act before asking about emissions optimism, climate concern, efficacy, and willingness to pay taxes vs after). Our PAP did not include these in the "good news" items that were analyzed in @fig-coef-plot, but it said we would analyze their effects along with other information treatments. We omitted this from the main body of the paper for presentational reasons: describing these treatments and discussing their effects requires space and makes the paper less focused on the effect of good news about climate mitigation.


```{r all-info-treatments, fig.height=7, fig.width=6.5}
#| label: fig-info-treatments-all
#| fig-cap: Effect of each information treatment, on average and by respondent pre-treatment factual pessimism

ira_on_efficacy <- for_info_treatment_plot |> filter(p.value < .05) |> filter(str_detect(treatment, "FL_39"))

#info_treatment_itx_regs |> 
#  mutate(tidied = map(reg, broom::tidy)) |> 
#  select(-outcome) |> 
#  unnest(tidied) |> 
#  filter(p.value < .05 & str_detect(term, "included:pessimism")) |> 
#  count(term)

for_info_treatment_plot |> 
  select(-treatment, -the_formula, -reg, -controls, -term) |> 
  mutate(group = "All") |> 
  bind_rows(for_info_treatment_itx_plot |> select(-interaction_label) |> rename(conf.high = ci_upper, conf.low = ci_lower)) |> 
  mutate(group = fct_inorder(group),
         this_alpha = ifelse(group %in% c("All", "Medium pessimism"), 1, 0)) -> for_info_treatment_plot2_full

for_info_treatment_plot2_full |> 
  ggplot(aes(x = estimate, xmin = conf.low, xmax = conf.high, y = fct_rev(treatment_label), col = group, pch = group)) + 
  ggstance::geom_pointrangeh(position = position_dodge(width = -.5)) + 
  geom_vline(xintercept = 0, lty = 2) + 
  labs(y = "Information treatment", x = "Estimated effect of providing treatment on standardized outcome",
       col = "Subgroup", pch = "Subgroup") + 
  scale_color_manual(values = c("black", "#F3AD1E", "#00A08B", "#ED2226")) +  
  facet_wrap(vars(outcome_label)) + 
  theme_bw() + 
  theme(legend.position = "top") + 
  expand_limits(x = c(-.32, .32)) + 
  geom_text(data = for_info_treatment_plot2_full |> mutate(n_label = str_c("n=", n)), mapping = aes(x = -.275, y = fct_rev(treatment_label), label = n_label, alpha = this_alpha), col = "black", size = 2.5, position = position_dodge(width = -.5), show.legend = F)

```

Telling respondents about the IRA is estimated to reduce respondents' sense of efficacy by `r round(ira_on_efficacy |> pull(estimate) |> abs(), 3)` standard deviations; the associated p-value (`r round(ira_on_efficacy |> pull(p.value) |> abs(), 3)`) is above the threshold we calculated *without* these extra information treatments (`r round(family_A_cutoff_info, 3)`), so we do not reject the null hypothesis. None of the interactions between pre-treatment pessimism and either the gas tax information treatment or the IRA treatment is significant at .05 even before adjustment for multiple testing.


```{r pretest-pessimism, eval = F}
# this code is not run, but can be used to get the summary statistics about knowledge in the pilot
pt <- qualtrics_load("./../good_news_bad_news_proposal/SSI3 final survey 2023 (SSD)_May 5, 2023_07.46.csv")
# how many more pessimistic than the correct answer? 
pt |> 
  mutate(pess_jobs = know_jobs != "Around 480,000, or four times as many",
         pess_solar = know_solar_cost != "The cost has gone down by around 80%", na.rm = T,
         pess_us_emissions =!know_co2_change %in% c("It went down by about 15%, to 5 billion tons", "It went down by about 30%, to 4 billion tons"),
         pess_something = pess_jobs | pess_solar | pess_us_emissions) |> 
  summarize(across(starts_with("pess"), mean, na.rm = T))
```


### Results using number of 'good news corrections' 

```{r}
#| echo: false
dat |> 
  filter(!is.na(number_possible_corrections)) |> 
  count(number_possible_corrections) |> 
  mutate(prop = n/sum(n)) -> npcs
```


Following our PAP, the analysis in the main paper focuses on the marginal effect of providing a correct answer to the quiz questions on our survey dealing with the cost of solar power, the size of the renewable jobs sector in the US, and the trajectory of US emissions. In @fig-coef-plot-a we show the same analysis as @fig-coef-plot, but here the treatment is not the number of correct answers provided but the number of "good news corrections". A respondent receives a "good news correction" when (i) she is provided with a correct answer to one of the three quiz questions, and (ii) her own answer to that question was less optimistic than the correct answer. (Thus this measure of the treatment recognizes that if someone already knew the right answer to a question, then providing the correct answer to that question does not constitute "good news" for that person and should have no effect.) The analysis underlying @fig-coef-plot-a is the same as the analysis underlying @fig-coef-plot with one exception: because assignment to treatment in this case is random only conditional on the number of "good news corrections" that the respondent could have received, we add a fixed effect for this number. (The number of possible good news corrections is 0 in `r round(npcs$prop[1], 3)*100`% of cases, 1 in `r round(npcs$prop[2], 3)*100`%, 2 in `r round(npcs$prop[3], 3)*100`%, and 3 in 2 in `r round(npcs$prop[4], 3)*100`%.) <!-- new --> 

The results are highly similar to those in @fig-coef-plot, with no significant average effects and significant effects in the same subgroups.

```{r fig-coef-plot-a, echo = F, warning = F, fig.cap = "Effect of 'good news corrections' on six outcomes, on average and across subgroups", fig.height=7, fig.width=8}
#| label: fig-coef-plot-a

key_estimate <- for_figure_1_a |> 
  filter(group == "High" & str_detect(interaction_label, "factual pessimism") & str_detect(outcome_label, "Donation")) |> pull(estimate)

for_figure_1_a |> 
  ggplot(aes(x = estimate, xmin = ci_lower, xmax = ci_upper, y = fct_rev(interaction_label), col = group, pch = group)) + # fct_rev(group))) + 
  ggstance::geom_pointrangeh(position = position_dodge(width = -.5)) + 
  geom_vline(xintercept = 0, lty = 2) + 
  labs(y = "", x = "Estimated effect of additional 'good news' correction on standardized outcome",
       col = "Subgroup", pch = "Subgroup") + 
  scale_color_manual(values = c("black", "#F3AD1E", "#00A08B", "#ED2226")) +
  facet_wrap(vars(outcome_label)) + # , scales = "free_x") + 
  theme_bw() + 
  theme(legend.position = "top") + 
  geom_text(data = filter(for_figure_1_a |> mutate(n_label = str_c("n=", n)), group %in% c("All", "Medium")), mapping = aes(x = .25, y = fct_rev(interaction_label), label = n_label), col = "black", size = 2.5)
```

### Testing for non-monotonic effects of additional pieces of 'good news"

An anonymous reviewer's comment raised the possibility that increasing the number of correct answers provided could have a non-monotonic effect on respondents' optimism about climate change mitigation. Specifically, the reviewer suggested that a respondent who is provided a single correct answer might infer that her (pessimistic) responses to the other questions are correct, making her more pessimistic than another respondent who is not provided any correct answers. To assess this possibility of non-monotonic effects, we regress each of our outcome variables on indicators for each possible number of correct answers (1, 2, and 3) and the same control variables as in the main analysis. The results, presented in @fig-non-monotonicity, show no evidence of non-monotonicity. <!--  new --> 

```{r}
#| label: fig-non-monotonicity
#| fig-cap: "No evidence of non-monotonic effects of additional information on our outcome variables"
#| fig-height: 6
#| fig-width: 5
#| out-width: "5in" 
#| out-height: "6in"

outcomes <- c("emissions_prediction_standardized", "concern_index_standardized", "efficacy_standardized", "wtp_taxes_standardized", "ira_response_standardized", "donate_amount_standardized")
outcome_labels <- c("DV: Optimism about\nfuture emissions", "DV: Concern about\nclimate change", "DV: Efficacy index", "DV: Willingness to\npay higher taxes", "DV: Support for\naggressive policy", "DV: Donation")
interaction_variables <- c("age_group", "ideology", "party_id", "pessimism_tercile", "concern_group", "efficacy_group")
interaction_labels <- c("By age (low=18-34,\nmed=35-49,high=50+)", "By liberal-ness (low=con,\nmed=mod,high=lib)", "By Democrat-ness (low=R,\nmed=I,high=D)", "By pre-treatment\nfactual pessimism", "By pre-treatment concern", "By pre-treatment efficacy")
controls <- c("as.integer(concern_cc_pre) + as.integer(lib_con)", "as.integer(concern_cc_pre) + as.integer(lib_con)", "as.integer(efficacy_pre) + as.integer(lib_con)", "as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing", "as.integer(concern_cc_pre) + as.integer(lib_con)","as.integer(concern_cc_pre) + as.integer(lib_con) + as.integer(hh_income) + hh_income_missing")

# regressions for average effects
tibble(outcome = outcomes,
       outcome_label = outcome_labels,
       controls = controls) |> 
  mutate(the_formula = map(str_c(outcome, " ~ factor(n_good_news_wo_ira) + ", controls), as.formula),
         reg = map(the_formula, ~ estimatr::lm_robust(formula = .x, data = dat))) -> ate_regs_fact 

ate_regs_fact |> 
  mutate(tidied = map(reg, broom::tidy),
         n = map_int(reg, nobs)) |> 
  select(-outcome) |> 
  unnest(tidied) |> 
  mutate(group = "All",
         interaction_label = "All") |> 
  filter(str_detect(term, "n_good_news_wo_ira")) |> 
  mutate(term = str_replace(term, "factor\\(n_good_news_wo_ira\\)", "")) |> 
  select(estimate, term, ci_lower = conf.low, ci_upper = conf.high, interaction_label, group, outcome_label, n, p.value) -> ate_regs2_fact

ate_regs2_fact |> 
  mutate(outcome_label = fct_inorder(outcome_label)) |> 
  ggplot(aes(x = estimate, xmin = ci_lower, xmax = ci_upper, y = fct_rev(outcome_label), pch = term)) + 
  geom_vline(xintercept = 0, lty = 2) +
  ggstance::geom_pointrangeh(position = position_dodge(width = -.35)) + 
  labs(x = "Estimated effect on standardized outcome\n(compared to zero correct answers provided)", y = "", pch = "Number of correct\nanswers provided") +
  theme_bw() + 
  theme(legend.position = "top")

```


